19

Please modify http://cdn.sstatic.net/robots.txt to allow the Internet Archive's crawler, ia_archiver.

Pages that were archived a few years ago include their stylesheets and appear correctly:

Pages that were archived more recently do not include their styles -- robots.txt does not allow it.

Because it doesn't use the CDN, MSO is still archived correctly:

Related: Jeff's post suggesting that people use Archive.org to archive posts.

0
10

We have removed the restrictions via robots.txt from http://cdn.sstatic.net which should allow crawlers to access the js.

6
  • Thanks. It seems to have the old file cached for now. Hopefully that won't last too long.
    – Jeremy
    Jul 21 '12 at 4:23
  • 1
    @jeremy, just in case you expect differently, I doubt allowing it now will also fix things for older pages? (The Internet Archive cannot tell what was the CDN content at the time the older pages were archived, but I guess you understand.)
    – Arjan
    Jul 21 '12 at 9:55
  • Ah, @jeremy, maybe your report actually will avoid bad content being archived, though not for a past date: "The Wayback Machine hasn't archived a capture for that URL. Here's a capture taken 0 minutes ago from the live web that will become part of the permanent archive in the next few months." Nice.
    – Arjan
    Jul 21 '12 at 9:59
  • @Arjan The archive sometimes matches up links from different years, so it's not implausible that they might apply the current stylesheet to the older pages. I'm not sure, though, and at any rate it probably wouldn't happen until the current version is in the permanent archive. The example post was temporarily undeleted to try this out, and it's now been re-deleted, so it may not be the simplest case to test with.
    – Jeremy
    Jul 21 '12 at 18:44
  • 1
  • How about http://i.stack.imgur.com/robots.txt? That's more stable, but still not as permanent as the Archive.
    – Jeremy
    Jul 23 '12 at 1:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .