TIL #001: pip cache does not work with Azure Feed

2023-04-22

til , pip , azure-devops

pip caches packages by their URLs, but Azure Feed’s short-lived redirect URL changes almost each time the package is accessed, preventing cache hit.

The problem can be illustrated with the following examples.

[1] Final package URL tends to vary:

[janiec@scarlet ~]$ for i in {1..10}; do curl -w '%{redirect_url}' -o /dev/null -s "https://$TOKEN@pkgs.dev.azure.com/$PACKAGE_PATH" | sha224sum; done
b2b61426a88bc429fa09291c14065b8c3015c3fc8f4467d23483a127  -
b2b61426a88bc429fa09291c14065b8c3015c3fc8f4467d23483a127  -
f9b8ca1d45a005bab0340d8574f78cedaa86482915d51c86d513ed22  -
f9b8ca1d45a005bab0340d8574f78cedaa86482915d51c86d513ed22  -
f87e170e51fdf1d59748f0de06e295ea7b131041b465affdb8aa6197  -
c763def60fe1d11c5a01481a8ee3416005e7bc89b20110d447abd490  -
7c20688899313789feee7c6840b5715d398b388f1b0578ecb19950d4  -
7c20688899313789feee7c6840b5715d398b388f1b0578ecb19950d4  -
f9b8ca1d45a005bab0340d8574f78cedaa86482915d51c86d513ed22  -
b2b61426a88bc429fa09291c14065b8c3015c3fc8f4467d23483a127  -

[2] Final package URL expires (after ~25 minutes in my recent test):

[janiec@scarlet ~]$ while true; do echo $(date -Is) $(curl -s -w '%{http_code}\n' -o /dev/null 'https://$SOME_ID.blob.core.windows.net/$SESSION_KEY'); sleep 30; done
2023-04-22T15:15:57+02:00 200
2023-04-22T15:16:27+02:00 200
2023-04-22T15:16:58+02:00 200
...
2023-04-22T15:38:25+02:00 200
2023-04-22T15:38:55+02:00 200
2023-04-22T15:39:26+02:00 200
2023-04-22T15:39:56+02:00 200
2023-04-22T15:40:26+02:00 403
2023-04-22T15:40:56+02:00 403

You can read more about it on Nathaniel’s blog.