Flyer screens

So we got that Android box running one of our flyer screens reasonably well, and ordered four more, and guess what? They all have exactly the same MAC address. No, that doesn't make our network lose its mind at all, why do you ask?

Seriously, who does that?

00:e0:11:22:33:44, if you were wondering. What a clever number.

Apparently you can only change this by jailbreaking them, which was an icepick we weren't exactly looking forward to trying on for size. Hooray.

Tags: , , , ,

39 Responses:

  1. Peter says:

    Chinese. Statistically, in a reasonable sample of MAC addresses of even consumer PCs, about 1-2% will be duplicates.

    • jwz says:

      Did you not notice that this MAC address, statistically, says one two three four?

    • Micah says:

      No they won't be. Vendors are assigned prefixes, and they have to not repeat addresses within their prefix. There are prefixes for local use (like 192.168/16, etc) but 00:e0:11 isn't one of them. It belongs to:

      00-E0-11 (hex) Uniden Corporation
      00E011 (base 16) Uniden Corporation
      2-12-7 Hatchobori,
      Chuo-ku Tokyo 104-8512
      JAPAN

      You can look all of them up here: http://standards.ieee.org/develop/regauth/oui/oui.txt

      Also, if 1-2% of MAC addresses were dups, it would be awful, no reasonably sized network would ever work.

      • Peter says:

        That's how it is supposed to work, yes, but not how it works in practice. Of course, exact results will depend on your sample. A large corporate network would have a different distribution than a thousand cheap consumer PCs independently bought at the nearest generic shop.

  2. David M.A. says:

    They obviously weren't planning on doing a lot of bulk business.

    • Ewen McNeill says:

      "Limit: 1 per customer" :-)

      Slightly more seriously, if you don't want to jailbreak them and are willing to "work around" this brokenness, connecting them to a VLAN-capable switch (one VLAN per device) trunked into a VLAN capable server should in theory cope better with the MAC duplication. That may be a little too much like Sysadminning Your TV though.

      Ewen

  3. Daniel says:

    That is the grossest thing. Either A) the network chips in these devices are so immensely cheap that they all had the exact same EEPROM load baked in or B) the people who built the finished device went through specific, deliberate software efforts to make this happen.

    Neither one is the least bit okay, but at least the hardware explanation is more understandable. Why bother slowing down your production line with a ROM-burning stage when you can just, y'know, not? It's awful, but at least it has a profit motive. Sadly I think it's less likely than some mislead 'engineer' manually setting the hardware address in an init script.

    Did you find info on fixing this via Google? I couldn't find anything specifically about this problem on this device, and I'm curious what the root cause actually is.

    • Christian Vogel says:

      Most of these SoC include the "Network Chip" on the processor die, there is no external eeprom only for storing the 6-byte mac address. Typically there are conventions to either have a few bits of config space (for serial numbers, cryptographic DRM-junk-keys, ...) in the processor, or some place in flash, alongside other bootloader-configs.

    • Zygo says:

      You assume that at this price point the vendor can afford to pour cash on extravagances like a whole separate uniquely-factory-programmed ROM to store just one NIC's MAC address. They can't.

      This kind of hardware has its host CPU load the MAC address into the NIC on every boot. A unique MAC address would be stored in a file on a filesystem or in some designated sector of the boot ROM. Some software then has to retrieve the MAC and pass it to the NIC driver before it will work.

      On better devices, the MAC is uniquely assigned at the factory; otherwise, the device may generate a random MAC on the first boot and save it for use in future boots. Of course, good quality random number generators are expensive, too, so don't bet against a high collision rate.

      On cheap devices, nobody bothers paying for all this extra work, so the NIC driver loads a constant MAC because that's good enough to test one device with 'ping'.

      One way to fix this is to load better software onto the device. Another way is install them all on separate networks. When we get boxes of identical eval boards, we put each one on its own USB-Ethernet NIC, and they all get their own private DHCP servers.

      • Daniel says:

        My bad, I was making the assumption that the NIC module included a one-time programmable ROM, which are inexpensive afaik but slow to program. It sounds like you know what you're talking about though, and I can totally see why they'd make it like this. Thanks for the insight.

        • Tim says:

          Since everything made today stores its firmware in flash memory, and you never need all of it for firmware, it's a no brainer to use six bytes of it to store a unique MAC address. Adding some OTP memory would be an extra cost.

          (That said, many modern network SoCs do include some on-die OTP -- the one I worked on did -- but it wasn't for MAC addresses. We also had some on-chip mask ROM containing the 'level 0' bootloader, responsible for loading customer bootstrap code from off-chip flash memory. This mask ROM bootloader had hooks to extend its functionality with code programmed into the OTP memory by the chip factory. This was insurance against bugs in chip logic or mask ROM which got past simulation and other forms of verification -- provided the bugs weren't too severe we'd be able to put workarounds in OTP. The OTP was necessarily very small, which is why we didn't just store all the level 0 bootloader code in OTP, but it was enough to store patches.)

          Getting back to unique MACs... Note that device firmware MUST be programmed by a station in the factory regardless of whether the MAC address is properly uniqified, otherwise the widget won't work at all. Furthermore it doesn't actually add any marginal cost (programming time, extra equipment, etc) to make MAC addresses unique. It's fairly trivial software in the firmware programming station, not rocket science at all. When you see this, it implies the vendor was outrageously over-the-top cheap (doesn't give a fuck about the customer cheap) in how they chose to set up their production line.

          • Zygo says:

            The widget will certainly work without a unique MAC address. There just can't be more than one on the same LAN (unless the device is a firewall and there is some kind of fancy transparent fail-over thingy going on).

            I can see you've never built a small production run from a contract manufacturer, and been given the option of a lower setup fee for flashing a totally constant firmware image.

      • Jason McHuff says:

        When we get boxes of identical eval boards, we put each one on its own USB-Ethernet NIC, and they all get their own private DHCP servers.

        Did your USB NICs come with unique MACs? Because I bought one on eBay very recently and found that it didn't, nor supports 100Mbs or USB 2.0 as described.

        Luckily, I only bought and need one, so don't have to override 00:e0:4c:53:44:58.

        • Zygo says:

          When we get the choice between the $4.74 NIC from a random eBay vendor and the $7.74 NIC from a supplier who knows what they're doing and values repeat business, we pay the extra three dollars.

          We might buy a handful of the $4.74 NICs and test them thoroughly first, but we'd have to be planning to buy hundreds of them. Even then, we wouldn't ship them to customers, install them in closets, use them for demos, or put them anywhere that we can't easily and inconspicuously swap them out if they turn out to be evil.

  4. Jeff Warnica says:

    That is the same MAC address as my luggage.

  5. jcv says:

    Some android devices helpfully change mac addresses on every reboot. That is also super helpful for networking. I guess the duplicates is a little more annoying though.

  6. Jeremy Wilson says:

    Ha ha! Fucking Brilliant!

  7. J. Peterson says:

    My guess is the SDK manual for the SOC they use has a section for configuring the MAC address.

    Which the sloppy developer never bothered to read and left it as the you-gotta-change-this "default" in the sample code he copy-pasted into the ROM.

    • Tim says:

      Having worked inside a network SoC design house I can tell you a large number of customers want you to provide working software for them, they can't do it on their own. They just build one of your reference system designs and use the matching reference firmware and maybe replace the bitmapped logos in the configuration webpage with their own.

      Bet you that this is one of those companies, and they didn't even bother making sure each unit got a unique MAC address in its firmware load.

  8. We actually have to use an ice pick in this process. The button to put it in "firmware update" mode is way at the end of the 1/8" A/V output jack, and has to be held down during the boot process.

  9. Bill Paul says:

    Apparently the CPU in this thing is the Amlogic S802. It has built-in support for 10/100 ethernet, USB and graphics along with video and audio hardware codec support. Unfortunately their web site doesn't include any reference manuals. (It just says "contact your local sales rep.") They're based in China, but their U.S. office is in Santa Clara, so maybe somebody could go there and rattle their chain.

    In reality though, it's Keedox that needs to have its chain rattled, because they're the ones who put this SoC in their product and created the software load for it. As has already been stated, the OUI 00:e0:11 belongs to Uniden Japan, and neither Keedox nor Amlogic has any business presence there, so clearly somebody pulled this address out of their ass. At least they didn't use 01:02:03:04:05:06. (Don't get me started.)

    The ethernet address for this thing is almost certainly stored in flash along with the OS image. In fact there's a good chance it's hard-coded inside the ethernet driver. It's up to the driver to program the address into the MAC's RX filter. You _can_ use a separate configuration EEPROM to hold the address, but that adds to your parts count. If there's already flash present, it makes sense to use that. I've seen eval boards come with both EEPROMs and flash, but production designs tend to be a lot more stripped down.

    Remember that window curtain control thinger you built a while back from some sort of ethernet-enabled project board, and how you complained that the user was forced to set a unique MAC address in their application program rather than there being one put on the board during manufacturing? It's the same problem here. Only in this case, the user isn't supposed to do that: the vendor was supposed to do it, and the screwed up royally by leaving that step out.

    Since Keedox makes these things, it's their responsibility to provision each unit that comes off the assembly line with a unique address. Assuming they only have flash as NVRAM, that would mean creating a special flash programming process to somehow give each unit its own address. But clearly splatting the same image on each board is just so much easier, and hell most people aren't going to buy more than one anyway, so who's going to notice, right?

    I think you need to complain to Keedox support. This is clearly their fuck-up. Unfortunately you may need to go several rounds with them to explain the problem to them in a way they can understand and convince them they need to fix it. You'll probably have to send all but one of the units back to them since they'll need to be re-flashed with a new firmware image to correct the issue.

    Or you could rant about it on the internet.

    • jwz says:

      Or I could ruin a minion's day by making him figure out how to jailbreak it and pull five different MAC addresses out of his ass.

      Which is what I did.

      • Otto says:

        So, 44 thru 48, then? Fuck it, why not.

      • Jason McHuff says:

        I have to ask: did you notice the MAC address on the first one before you bought the others and inflicted that pain? I mean, it's not like the address looks totally random like the one the model of USB Ethernet adapter I bought uses.

        And why did you need to assign five addresses? What was wrong with using the supplied one for one of them?

        • joe luser says:

          because now he can buy at least one more without having to re-learn the jailbreaking process.

  10. gryazi says:

    Hm, I have a nameless MK802 somewhere, the kind with the horrible wireless range. Will have to find out if it's Everyone once I figure out where it went.

  11. thx1138 says:

    Had dupe MAC addresses on two expensive "enterprise grade" Nokia firewall quad NICs once. Deploying this pair turned into a literal "cluster fuck."

    • Dusk says:

      For a while, Sun had a unique interpretation of MAC addresses that resulted in multi-NIC servers having the same MAC on every interface.

      • Wow. That’s bananas. The sysadmins must’ve been furious.

      • Rui Carmo says:

        Yep. I fondly recall the time when the Sunscreen firewall we had booted up and set the same MAC address on all interfaces, confusing the heck out all four of our Cisco switches for an hour.

        Every morning or so.

      • gryazi says:

        That sounded sort of familiar and I just had to go look up the logic...

        Apparently it went like 'Well, each interface is going to a different network, so you who cares [and you can identify the machine on each of them]?'

        • Bill Paul says:

          Oh god, the multi-port "happy meal ethernet" cards. I remember those.

          Yes, you only fail in that case (multi-port NIC with same MAC on all ports) if you connect two or more ports with to the same ethernet network. And most of the time, that doesn't happen: how often do you plug two ports from the same host into the same switch/hub/VLAN? Maybe you'd do it for channel bonding or failover. But the logic there is that you're going to pay extra money for a special NIC for that purpose (which is really just like any other multi-port NIC, but with special driver software, a higher price tag, and maybe a fancy name with the word "turbo" in it).

          Of course, that was before IPv6 came along. Having multiple interfaces with the same IPv6 link local address can't be much fun.

          Incidentally, the multi-port Intel PRO/1000 NICs (and this includes both the older dual port PCI-X NICs with the 82546 chip as well as newer dual and quad-port PCIe ones with the 82575, 82576, 82580 and i350 chips) actually use one EEPROM (or flash) device for all ports. Each port is a separate PCI function, so to the host it just looks like there's multiple cards plugged in, but internally each function shares access to the same NVRAM. This means there's really only one MAC address programmed at the factory for the whole card, just like the Sun multiport cards.

          However, there's a "function ID" field in the device status register which the driver can use to figure out which port it's working with, and Intel's driver software uses this value to tweak the last bit (or in the case of 4 port cards, the last 2 bits) of the station address read from the NVRAM to ensure that each port's address is unique.

          I have no idea why Sun didn't do things the same way. Maybe they were trying to avoid using up their allotment of addresses too quickly. In any case, I think most sysadmins resorted to using "ifconfig hmeX ether xx:xx:xx:xx:xx:xx" to tweak the MAC addresses manually if it became necessary.

  12. Ronald Pottol says:

    Wow, my mind is blown. I mean, I worked at a place that used 1.x.x.x for our private address space for our WiFi, and while those people were gone, I knew that there would be no limits to the horrors I might later find, and I was right.

    But this is such an epic fail, you cannot have two of these on the same network, so you'd think someone would notice and fix it, besides, you are breaking the spec. Random would still be wrong, but at least it would mostly work. Gah. The horror, the horror....

  13. Luke says:

    On the bright side, at least those MACs have a valid (but perhaps not "legitimate") prefix - I bought some network cards for a PC at uni and they came with invalid MACs. My uni's network does not accept MAC addresses that don't belong to a vendor - you can't register them, so they'll just get stuck on the registration VLAN.

    Fortunately they were at least all unique...

  14. Luke says:

    On the bright side, at least those MACs have a valid (but perhaps not "legitimate") prefix - I bought some network cards for a PC at uni and they came with invalid MACs. My uni's network does not accept MAC addresses that don't belong to a vendor - you can't register them, so they'll just get stuck on the registration VLAN.

    Fortunately they were at least all unique...

  15. NB says:

    Part of the blame falls on the IEEE for hiking the prices of MAC space allocations in recent years, apparently enough to cause cheaper manufacturing outfits to skip acquiring a range altogether.