Dear Lazyweb,Can anyone give me working instructions on how to mitmproxy the Instagram app, from macOS, using either a real iPhone, or an emulated Android or iOS device?
When I proxy the iOS version, Safari and Twitter work, but Instagram says "Couldn't refresh feed" or sometimes "no Internet connection". It's detecting the proxy somehow. Maybe cert pinning?
It appears that there are people out in the world successfully proxying it on Android, but I can't figure out how to proxy the Instagram app in an Android emulator. There are instructions to make proxying work with non-Chrome apps, and those involve doing "adb root", but that doesn't work on Android installs that have Google Play on them, and without Google Play, how do I get the Instagram app installed in the emulator at all? (Note, I know nearly nothing about Android.)
You can usually find extracted .apk's for most apps on the interwebs. APK Mirror is a good choice for being reasonably confident it is the real thing and not pumped full of Chinese malware:
https://www.apkmirror.com/apk/instagram/
Once you have the apk, you can use "adb install" to push it to the emulator.
You might need to set up an ARM image in your emulator (instead of x86) since the extracted images are for ARM ABIs. It might also not matter if the app doesn't have any native libs (I don't know for Instragram)
How do nightclubs not owned by former netscape engineers get anything done
They hand over the business to corporate booking agencies, pick their vibe and offerings off the retail rack, and generally give up on being more interesting than, say, a combination Pizza Hut and Taco Bell?
Just guessing....
See if you can download the .apk for instagram on APKMirror and replace the CACert in the APK with your mitm one.
Use APK Studio to open the .apk itself, then look around for any CA cert. If you're lucky, you'll easily be able to delete the old CA, then put in yours.
If you click 'Reply' on a post, refresh the page, then put your comment in the now non-reply field, it still posts the comment as a reply to someone.
I took "delete the old CA, then put in yours" as a poetic description of the war on fun that hurts nightclubs, so it works out OK anyway.
I think I'm almost there, but it's not working. I did this:
1: Android Studio, upgraded everything.
2: Created new emulator, Nexus 5 API 28, "Google API" not "Google Play"
3: cd ~/.mitmproxy/
4: openssl x509 -inform PEM -subject_hash_old -in mitmproxy-ca-cert.cer | head -1
5: cp mitmproxy-ca-cert.cer c8750f0d.0
6: emulator -avd Nexus_5_28 -writable-system
7: adb root
8: adb shell "mount -o rw,remount /"
9: adb reboot
10: adb push c8750f0d.0 /system/etc/security/cacerts
11: adb shell "chmod 664 /system/etc/security/cacerts/c8750f0d.0"
12: Verify that it is a system-trusted cert by looking at Settings / Security & location / Encryption & credentials / Trusted credentials.
13: Preferences / Wifi / "AndroidWifi" / pencil-icon / Advanced options: Proxy: Manual 10.0.1.2 8080
14: Open a page in Chrome: it goes through the proxy. Yay.
15: adb install -r com.instagram.android_123.0.0.21.114-188791703_minAPI23\(x86_64\)\(nodpi\)_apkmirror.com.apk
16: adb install -r com.twitter.android_8.24.1-release.01-18241001_minAPI21\(arm64-v8a,armeabi-v7a,x86,x86_64\)\(nodpi\)_apkmirror.com.apk
But Twitter can't log in, doesn't proxy; Instagram can't log in, doesn't proxy.
Can you try CharlesProxy instead of mitmproxy to see if proxying works at all?
https://www.albertomoral.com/blog/get-data-from-instagrams-private-api
https://www.charlesproxy.com/download/
There's like no chance that mitmproxy wouldn't at least log a connect(). Also Chrome works.
I ran Instagram through CharlesProxy on iOS 13, and it seems like Instagram isn't trusting the installed certificate, here's a screenshot, and I've pasted the full errors below. I installed the Charles certificate per their guide, https://www.charlesproxy.com/documentation/using-charles/ssl-certificates/
<img src="https://i.imgur.com/gQBUk4I.png" /
Failure
SSL handshake with client failed - The client supported protocol versions [TLSv1.3, (D)TLS--5.26] are not accepted by server preferences [TLS12]
Notes
You may need to configure your browser or application to trust the Charles Root Certificate. See SSL Proxying in the Help menu.
TLS Alert Code
protocol_version (70) - The TLS protocol version proposed by the client is not supported by the server
so have fun...
Cute. Yeah, so they refuse to talk TLS 1.2 or earlier, knowing that their servers all can do TLS 1.3 so if you can't then you must be an impostor.
This is extra confusing for a naive TLS 1.2 server because TLS 1.3 clients don't say TLS 1.3 where it expects. Middleboxes freak out if you say you speak any version they don't know about even though that's the whole point of version negotiation, so an actual TLS 1.3 client says "Hi, I'm resuming a TLS 1.2 connection which you'll remember from earlier, also I support a cool new optional extension FlyCasualThisIsReallyTLSv1.3 with parameters that look exactly like ECDHE key exchange" and then a TLS 1.3 server goes "Right you are, TLS 1.2 resumption, I too know FlyCasualThisIsReallyTLSv1.3 with parameters that look like my half of a key exchange" and then like in Portal 2, "Surprise! We're Doing It Now" everything after that is encrypted and the middlebox figures this must be an old session it has previously allowed through so no need to intervene. If you as server don't understand TLS 1.3 you go "Um, what? No we've never spoken before, maybe I was rebooted. Anyway TLS 1.2 is fine but let's start from the beginning as usual and I don't understand that fancy new extension" and that incurs the round trip cost we're used to for TLS setup.
The "real" clients are definitely key pinning. So you will need to obtain "cracked" clients which patch out the pinning checks or they'll abort even if you are able to MITM the connection.
Ok but I'm able to proxy the iOS Twitter client, which means it's not pinning. It would be weird for it to be pinning on Android but not iOS, wouldn't it?
I've yet to see any proxy hits from an app other than Opera, so I don't know that proxying is working with any app. Not sure how to test that.
It is definitely possible that explicitly hooking up your proxy (in Android Studio?) in step 13 causes apps which don't want to go through a proxy to ignore it completely and just try to connect directly or fail.
Silently MITMing their networking (so that they can't opt out though they might still detect you and fail later) is a bunch more work for which I have no pre-baked instructions. The least invasive thing that could work for that is to provide lies for DNS answers, claiming your mitmproxy is the real address for any server and much more drastic is to just force all the packets to go to mitmproxy even though they have the wrong destination, then "fix" packets coming back out to show the expected source.
Well, according to this, mitmproxy doesn't support TLSv1.3. And there's this which says "Most of our dependencies, openssl, and their Python libs, should support it though". MacPorts gave me mitmproxy 3.7, but I tried the git head version, 6.0.0.dev, with Python 3.7.6, and Android Twitter and Instagram still don't proxy. I don't know how to tell whether 6.0.0 thinks it supports TLSv1.3 or not.
But again, proxying iOS Twitter works fine with mitmproxy 3.7.
Does it seem plausible that the iOS version of the Twitter app works with TLSv1.2 but the Android version requires TLSv1.3?
Does it seem plausible that the Android version pins certs but the iOS version does not?
Maybe?
For the FB properties:
https://engineering.fb.com/security/fizz/
"We have deployed Fizz and TLS 1.3 globally in our mobile apps, Proxygen, our load balancers, our internal services, and even our QUIC library, mvfst. More than 50 percent of our internet traffic is now secured with TLS 1.3. We also deployed zero round-trip resumption (0-RTT) data, the newest feature of TLS 1.3, with our mobile apps."
which also led to this:
https://plainsec.org/how-to-bypass-instagram-ssl-pinning-on-android-v78/
that's a year old and hocusy pocusy.
for twitter? maybe the android team decided fizz was cool, and the iOS team didn't include using external libraries in their KPIs?
I also tried Fiddler but I can't figure out how to get it's certificate onto the device, iOS or Android. The instructions have me downloading an EXE file, yay. And the magic URL doesn't work.
Ok, I got the cert installed and trusted on iOS, and Fiddler is decoding HTTPS traffic from Safari, but for any apps, it seems to be causing HTTP CONNECT tunnels instead of GET, with no decoding. Their documentation sucks.
mitmproxy does log connections but only silently to its Event log, if you aren't looking at the Event log you won't see a connection that mitmproxy didn't end up proceeding with.
How do I see this event log?
I'm seeing it proxy when using Chrome, XScreenSaver, Google Maps, and the Youtube app. Still not working with Twitter or Instagram, though.
In the command line mitmproxy it was shift-E to see the event log.
e.g. if I just telnet localhost $MITM_PORT then that shows nothing on the normal mitmproxy UI but in the event log I get a line saying somebody connected, and, when I disconnect the telnet, that they disconnected.
Ok, with both versions of mitmproxy the Android emulator and iOS hardware both show me Events but not Flows from Twitter and Instagram.
Yes this means that suddenly I'm no longer able to proxy iOS Twitter either, when I definitely was able to a couple days ago. WTF.
(Also I tried adding SSL_version => 'TLSv13' to my perl code, and that was recognized but didn't help.)
There is Frida, which allows MITM-like functionality by hooking the SSL library calls in the application instead of trying to proxy the network connection. At hack.lu 2019, Axelle Apvrille showed how she used Frida to trace HTTP/S functions in Android apps (including Instagram around 12:00 in the video) in a way that was much more exact than wireshark MITM of the packet.
Being able to write python hook for specific library function calls would allow you to drive the application as well, if that is your end goal.
TLS 1.3 support in mitmproxy appears to be implemented in this not-yet-merged pull request: https://github.com/mitmproxy/mitmproxy/pull/3692
When I apply that patch it dies with:
Requirement.parse('cryptography>=2.8'), {'pyOpenSSL'}
Hand-hacking the version number doesn't fix it. But also this is confusing because:
port installed | grep cryptography → py37-cryptography @2.8_0 (active)
I am in a twisty maze of python virtual environments, all alike.
Edited the version number in setup.py and then it seemed to install 2.8 but then dies with:
module 'lib' has no attribute 'SSL_CTX_get0_param'
This could be caused by a couple of things, jwz.
Most likely, I'd suggest it's this: https://android-developers.googleblog.com/2016/07/changes-to-trusted-certificate.html
Since Android 7.0, certain apps can basically ignore the certificates you add to the store - it's really irritating. Try installing Instagram in an Android 6 emulator (Genymotion works).
They might also have certificate pinning - I've never looked at the instagram app before, but it seems plausible. You could use something like Frida or Xposed to hook any cert-related functions and break the pinning, though.
Fails in the same way when using API 23 (Android 6.0 Google APIs, Nexus 5).
Try another way.
Requirement:
1. Android Emulator with OpenVPN client installed.
2. Server with OpenVPN and mitmproxy
Idea: MITM between OpenVPN Server and Internet. Connect OpenVPN client to your server, it will route all traffic from Emulator to OpenVPN server. You can replace OpenVPN with any VPN server that can be proxy.
No fix for this?
I never figured out how to MITM any official Instagram app, no.