Dear Lazyweb,
I have several Blackmagic MiniRecoder SDI → Thunderbolt dinguses. Every now and then they lose their minds and stop producing video unless unplugged and re-plugged.
Two questions:
- How can I reset it without physical access? Is there some way to reset the whole thunderbolt bus, or to power-cycle all devices attached to it?
I have also noticed that sometimes it gets weirdly finicky about the cab.e Sometimes reversing the cable is necessary to make it work again. I have a stack of these fucking cables, some brand new, and sometimes one works and another doesn't; and then they trade, and the old one is the only one that will work.
- How can I detect the thing having gone into the bad state?
When it is in the bad state, the "Blackmagic Desktop Video / Desktop Video Setup" app consistently says "No desktop video device connected" rather than listing the device name. So what I want to know is, how do I get that status result from a command line tool?
When it is in the bad state, "system_profiler SPThunderboltDataType -json" still lists the device. At some point I had believed that if link_status_key was 0x1 or 0x2, things were good, otherwise they were bad, but that is not good enough, apparently, since right now I'm seeing 0x2 but it's not working.
Here's the output on a Mac mini i5, 10.15.7, with a MiniRecorder in a working state:
{
"SPThunderboltDataType" : [
{
"_items" : [
{
"_name" : "UltraStudio Mini Recorder",
"device_id_key" : "0xA129",
"device_name_key" : "UltraStudio Mini Recorder",
"device_revision_key" : "0x1",
"receptacle_upstream_ambiguous_tag" : {
"cm_asn_key" : "C4M050204F5F797AG",
"cm_version_key" : "1.0.16",
"current_speed_key" : "Up to 10 Gb/s x1",
"link_status_key" : "0x2",
"pm_version_key" : "0.2.5",
"receptacle_status_key" : "receptacle_connected"
},
"route_string_key" : "1",
"switch_uid_key" : "0x0004000000130970",
"switch_version_key" : "7.2",
"vendor_id_key" : "0x4",
"vendor_name_key" : "Blackmagic Design"
}
],
"_name" : "thunderbolt_bus",
"device_name_key" : "Mac mini",
"domain_uuid_key" : "73874C54-62C9-C753-BD76-E1EA186DD0DD",
"receptacle_1_tag" : {
"cm_asn_key" : "C4M050204F5F797AG",
"cm_version_key" : "1.0.16",
"current_speed_key" : "Up to 10 Gb/s x1",
"lc_version_key" : "0.12.3",
"link_status_key" : "0x2",
"receptacle_id_key" : "1",
"receptacle_status_key" : "receptacle_connected"
},
"route_string_key" : "0",
"switch_uid_key" : "0x0001000D194BC520",
"switch_version_key" : "23.4",
"vendor_name_key" : "Apple Inc."
}
]
}
And here it is on a Mac mini i7, 12.6, with a MiniRecorder in the bad state:
{
"SPThunderboltDataType" : [
{
"_items" : [
{
"_name" : "UltraStudio Mini Recorder",
"device_id_key" : "0xA129",
"device_name_key" : "UltraStudio Mini Recorder",
"device_revision_key" : "0x1",
"mode_key" : "thunderbolt_one",
"receptacle_upstream_ambiguous_tag" : {
"cm_asn_key" : "C4M310400W1F798A5",
"cm_version_key" : "1.0.16",
"current_speed_key" : "Up to 10 Gb/s x1",
"link_status_key" : "0x2",
"pm_version_key" : "0.2.5",
"receptacle_status_key" : "receptacle_connected"
},
"route_string_key" : "3",
"switch_uid_key" : "0x00040000001FD690",
"switch_version_key" : "7.2",
"vendor_id_key" : "0x4",
"vendor_name_key" : "Blackmagic Design"
}
],
"_name" : "thunderbolt_bus",
"device_name_key" : "Mac mini",
"domain_uuid_key" : "C6AAD605-5E79-5159-B46F-80B40E12FDBA",
"receptacle_1_tag" : {
"current_link_width_key" : "0x1",
"current_speed_key" : "Up to 20 Gb/s x1",
"lc_version_key" : "0.14.0",
"link_status_key" : "0x7",
"receptacle_id_key" : "1",
"receptacle_status_key" : "receptacle_no_devices_connected"
},
"receptacle_2_tag" : {
"cm_asn_key" : "C4M310400W1F798A5",
"cm_version_key" : "1.0.16",
"current_link_width_key" : "0x1",
"current_speed_key" : "Up to 10 Gb/s x1",
"lc_version_key" : "0.14.0",
"link_status_key" : "0x2",
"receptacle_id_key" : "2",
"receptacle_status_key" : "receptacle_connected"
},
"route_string_key" : "0",
"switch_uid_key" : "0x000100150242A160",
"switch_version_key" : "26.1",
"vendor_name_key" : "Apple Inc."
}
]
}
Previously, previously, previously.
It is likely the unit overheating. Blackmagic are great but not really designed for 24/7 uptime unless you make sure they have good heat disipation, even the external self-contained boxes like you have. We constantly fight with cards and units just going into a failed state until power cycled, which usually takes enough time to cool off enough to reset. If possible, add some kind of additional airflow to the unit. As for cables, sometimes the same thing happens. Thunderbolt cables do have microcontrollers in the connectors, they are not straight passthrough wiring, and can “fail” temporarily from thermal stress.
Some Thunderbolt cables are really active cables, they have a repeater/redriver/retimer chip in-line with the high-speed signalling.
Other Thunderbolt cables are passive: they have a chip for identification but this is really only used to provide about under a hundred bits of information back to the PC about what the cable is; the signalling is still just through unadulterated copper. The passive ones will be short, and I would recommend that if you have reliability problems and suspect it's "the cable"
Also, the contacts in cable connectors oxidize. The act of unplugging/replugging wipes some of that oxidization off, and may make them "work again" - I find that after 1-3 years this becomes measurable. Note that I only say "measurable" but these systems have a significant amount of margin in them, so it's usually OK. Worth being aware of though. Humidity makes it worse. If I am truly depending on a cable, it goes in the trash after a few years.
When I started suspecting the cables I bought a few new ones, and from Apple, so some of these cables have single-digit hours of use on them. They are short cables.
Overheating is a plausible theory, but for what its worth, the devices are only barely warm to the touch.
An overheating chip won’t necessarily heat up the outside of a plastic case very much, especially if there’s an air gap between the chip/heatsink and the plastic case.
Across the universe of things with different names but USB-C shaped plugs and sockets, I've had generally poor reliability and durability. Extra-expensive premium brand wires seem to help, but increasingly I think that form factor will turn out to have been a historical mistake.
Are you still open to the idea of crowdfunding a poe/IP based option? It _could_ also be analog cameras over cat5e (they can easily do 200-300meter runs) down to a single DVR that can then output all streams in digital form simultaneously. Or switch at the DVR directly and get a single, switched up stream you could feed into ovs.
Asking cause I just did a setup where individual cameras go for 30-50 bucks a piece (this would be 1080p). The 16-input DVR is another 300 or so dollars.
Cameras can all be centrally powered and a single cat5 cable will let you run 2 cameras + power or 4 cameras if powered locally instead from the remote.
Sorry for answering the wrong question...
The first step down that path would be for someone to say, "Here is a spec for a system that will meet your requirements reliably. Now all you need is fifty grand."
Nobody has ever shown me such a spec.
In fact, the last time I brought this up, several people said "there's nothing wrong with SDI, it is designed specifically for the distances you are using, so something else is wrong."
Let me see if I can spec something up. What are the requirements you want? 1080p/30fps streaming? Is 15fps acceptable? Audio is still going separately I asume ? (Due to YouTube content strike)
I used to live literally next door to dna... Loved the live stream as it was a great way to know when the club was full ... Also the Livestream made for an awesome "pre game" at home.. especially for bootie mashup. I'm now living in Argentina. But I have, right now, a 16 camera system I'm installing so I could test a few things out
Right now, the DVR can record 16 cameras and stream each individuall as rtsp streams which I can watch in VLC so I'm almost sure OBS would work.
The only issue with my recorder right now is that at 1080p it only does 15fps. If the cameras are 720p then it can do 30fps.
The cameras are 30fps and can do color down to 0.01lux. they also have IR illuminators as well for full darkness
Each camera is around 30$. The DVR is around 300.
You also get a "free" security system out of the whole thing (if you want.. you can just skip installing a hard drive on the DVR and then it just stream)
The current system is:
The thing that differs between this system and things that other people commonly build include:
Great! thanks. Let me see what I can come up with!. I hadn't looked at the stream since a little while I so I was stuck with the split audio/video you had at some point in time (with youtube on "mute"). I see that now its all a single stream with audio...
The output to the bar could be handled with the local hdmi output of the recorder.
The good thing about using security cameras, IMO, is that they are meant to be ran unattended forever.. so that should help with stability... ideally!
I think the biggest challenge, ATM, would be good audio sync.
These SDI cameras we have are "security cameras" and they are not reliable.
I don't have this exact hardware, so take this with all the grains of salt. But I used to run audio through a blackmagic decklink studio and it would sometimes stop functioning. Unloading the reloading the blackmagic kernel extension would get it working again. For convenience I automated it via an Alfred shortcut that ran this code:
Why I ran a couple of shell commands via an applescript interpreter is lost to time.
Nope, reloading BlackmagicIO does not make it come back to life. And attempting to reload IOThunderboltFamily, BlackmagicUsbIO or Blackmagic_Multibridge_Driver fails.
Oh yeah, also four or more of the input channels on our Blackmagic Videohub 20x20 have died. So if you've got one of those lying around that you'd like to donate, that would be great... These things are still going for like $2K used on eBay. (Yes, the 12x12s can be had for cheaper, but I need at least 18x2.)
Regarding: "or to power-cycle all devices attached to it?"
That part is doable via a managed PDU (Power Distribution Unit), they come in various form factors and price points, but 42U rack mount intended devices with RJ45 10/100 Ethernet and SNMP for stats and usually some sort of CLI access via SSH (or telnet, yikes! Please use a separate management network with an encrypted tunnel) I seem to recall being $700+ on the used after market over a decade ago? Cheaper options exist, in a pinch some UPSes may even function in a similar role.
As for the rest of it, I can't say. The last piece of Black Magic Thunderbolt gear I owned seemed incredibly unreliable and Thunderbolt is powered so the overheating issues others mentioned are exacerbated over time. I also haven't been paid to work on video related kit since around 2014, but it seems as if since then many of the "cool kids" were migrating away from (HD/12G)SDI to NDI and oh look, it's that whole "converged" network architecture that so many used to opine for, it just costs a bunch of new gear to get there. :-/
Uh huh. Gonna need some clarity on how a PDU -- which I have -- will help me reset a device that is powered via its Thunderbolt connector.
My apologies. I misunderstood and thought you were referring to a "hail Mary" type scenario where power cycling whatever the host device(s) providing power to the Thunderbolt peripherals might help.
The only other thing that comes to mind is I know macOS/OS X will treat some Thunderbolt devices as network capable in which instance, maybe you would be able to use ifconfig enX (or whatever the enumerated device is) down/up. That seems pretty unlikely with video gear and I long since relegated my Blackmagic Thunderbolt capture device into storage so I can't even test that use case without a bit of digging.
Despite the technical frustrations, I hope you are able to enjoy the Front 242 show tonight! I seem to recall they may have provided some inspiration to you going down the whole DNA Lounge path in the first place.
So I went in to fuck with it in person, and the Mac mini was behaving as if maybe it had just decided "I don't do Thunderbolt any more." It wouldn't work with any of the MiniRecorders, but they all worked on another Mac. So I table-flipped, and got a brand new M1 Mac mini from Best Buy. And a fifty dollar Thunderbolt 2 to Thunderbolt 3 cable, because Connector Conspiracy. And guess what? Even with this Apple-approved cable, the new Mac won't talk to the old MiniRecorder. So now in addition to the new Mac, I have to buy a new Thunderbolt 3 MiniRecorder, which, in addition to being a stupid waste, means that the recorders are hardware-paired to their Macs and can't be swapped.
So that's all going great.
$50 US for a cable, and not even something pitched at 'audiophiles'. Yikes
It's an active cable and can carry 20gbps, though I may have gone for the $35 bidirectional TB2/miniDisplayPort male to TB3/USB-C female adapter.
K3n.
What's the word that's not schadenfreude, but means "enjoying someone else's tech misery because, thank fuck it's not me, but also hoping the poor schmuck manages to solve his problem because, (a) he's a nice guy and (2) I really want to know the solution".
That's me.
Thanks for the good time on the Internet.
"skootenfroody"
Connector Conspiracy Part Two:
No audio input means I have to consume another USB port with an external audio interface!
USB-C is so much more convenient because the connector is smaller, here, we'll demonstrate that by giving you fewer of them.
And in 2009:
Prediction: 1 USB A and 1 USB C in 2025.