Modifications for improved Non-Amazon S3 service providers, primarily allowing for the correct download of attachments. Tested with Backblaze, Wasabi, DigitalOcean, iDrive, etc. Service Status ACP Dashboard widgets for Backblaze, DigitalOcean, and Wasabi.
New Region Setting on S3 Storage Settings:
If you are using DigitalOcean or Wasabi, toggle the new "set region" to on, enter the region you have set for your service, and save. If Invision Community displays a pop-up modal warning you that you are making changes and it will init a move process, choose the option stating I already moved the files. All this setting does is aid the IPS S3 code which defaults to Amazon us-east when manipulating S3 things when it cannot detect a region correctly and it needs to. If you get to choose and set a region for your bucket, enter it here. This is NOT essential, but it could be useful if you run into problems (provides certainty)
ACP Dashboard S3 Service Monitors:
Available for Wasabi, DigitalOcean, and Backblaze. If there is a hew and cry for Vultr that can be written out as well for a future release. Widgets display region and occasionally other service-specific availability (online, impacted, down) along with service alert and status text if available. For services with a history function regarding alerts you can configure how much is available in the widgets in the S3WB settings. In most cases the alert text is truncated with an expand toggle to see the rest if needed. See screenshots. Service logo buttons link to what would be your account or configuration pages at those services. Willing to add additional monitors for other providers if there is a need, and they have something to get on that front (iDrive does not).
Code-modifications allowing the correct downloading by users of attachments and Invision Community Downloads file downloads:
Simplified header information mostly that fixes signature-breaking structure present in the code. Note that this is due to alternative S3 providers not being fully compatible with the Amazon S3 calls IPS makes by default. For Backblaze I bypass all S3 calls and instead utilize the Backblaze API system for users to download files.
Why:
The built-in S3 file handling system in Invision Community is programmed exclusively for Amazon S3. Although this DOES work with alternative S3 providers, most of these providers have quirks of their own that break full compatibility with what IPS provides. I wrote out a hacky plugin years ago to address some of these things, and even though that plugin still works, the actual plugin itself has not been updated for at least three years and was last flagged as compatible for the 4.5x series of Invision Community. We're a long ways away from that time, both with the IPS codebase and the S3 providers themselves. As we near the end of the 4x line I thought it was time to get this all updated to something relatively stable and more useful than a plugin.
Testing:
I have tested Backblaze, Wasabi, DigitalOcean, iDrive, and many months ago Vultr (don't hold me to that one but it should be fine I just don't want to spool off $5 to test again). I should point out I DO NOT make any alterations to the majority of the internal IPS S3 code. Testing included:
Moved attachment files from LOCAL STORAGE to ALT S3 SERVICE
Uploaded a new attachment
Tested downloads worked correctly for attachments
Moved attachment files from ALT S3 SERVICE to LOCAL STORAGE.
Confirmed files were intact and downloadable.
Moved Invision Community Downloads files from LOCAL STORAGE to ALT S3 SERVICE
Uploaded a new Downloads file
Tested downloads worked correctly for Downloads files
Moved Invision Community Downloads files from ALT S3 SERVICE to LOCAL STORAGE.
Confirmed files were intact and downloadable.
Testing was usually with 10 to 20 meg PDF files and of course the usual embedded images in posts (which are attachments) and all that was fine. If you are uploading massively large files it is up to you to investigate whether 100meg or larger files work correctly with these systems. This gets into the realm of chunking and your PHP/Apache settings and the like. Not my problem and again, nothing to do with this code at all.
You'll also notice I did not test with CDNs. Way beyond the scope of any of this. The old plugin is working fine with CDNs if configured properly; you'll have no problems here either - again, the PRIMARY FUNCTION of this is to patch up attachment downloads and makes ZERO CHANGES to anything else.
Lastly, transfers from one S3 service directly to another S3 service were not tested. Again, nothing to do with this application. If you have a substantial amount of files to move between storage areas, you are better off using other dedicated tools and then flagging changes to the file systems as "I already moved the files". Backups are not a bad idea either...
The above testing was two fold: to confirm the attachments downloaded with S3 Workbench, and that the services nominally worked. The former was required; the latter is a courtesy.
Requires:
cURL for Backblaze (The 4x line still allows fallback to sockets if you are running an ancient cURL version but the BB stuff is all cURL).
You should know how to configure S3 in general.
Invision Community Downloads:
If you are using this IPS application, S3WB does work with those file downloads (and unlike the old plugin, no longer FORCES them to be public files) but if you absolutely MUST have those file downloads hyper-secure be sure to use your local storage or Amazon S3. If one of these downloads files is an mp3 it will probably load in the browser directly when the user grabs the "file". This is all the same code to generate the URL as regular file attachments, but it appears to be more sensitive for "reasons" that are not worth my time to track down. I know why this happens, but am unsure of its uneven application (pdf's I would image would load in the browser the same but they do not).
Additionally, some of the alternative S3 services out there do not do file-level access control (e.g. flagging a file as private) but instead do public and private buckets. I'd just point out that once a user has one of your "private" files, they can do what they want with it - including sharing it it, re-uploading it, and so on, and there is nothing you can do about it. So, again, if this is all hyper-important to you, just use local storage or Amazon S3, otherwise, really, it's fine.
Service Specific Notes:
All the settings below, including bucket names, are obscured/fake but the configurations are correct. Provided as a courtesy.
Backblaze:
S3WB uses the Backblaze API, not the Backblaze S3 implementation, for file downloads. An additional change is dropping the amz-acl header calls when a url is doing something with Backblaze - the service DOES NOT WANT ANYTHING TO DO WITH acl. Works great and I'd be fine recommending these guys. Crazy stock price crash since listing, though prob a buy opportunity right now (Mid-June 2023 it's $4 something, it's $5 something mid-July, expected to do $9 something).
bucketname: muhbucket
endpoint: s3.us-west-004.backblazeb2.com
access key: abcdefg1234
secret key: abcdefg1234
set region ON
region: us-west-004
Wasabi:
Solid. The one that started all this mess years and years ago. I'd consider them stable now; in the 99%'s, which is kilometers (or miles if you prefer) better than they were on initial launch. I still have a $5/month account with them (that's how early I got on the wagon).
bucketname: muhbucket
endpoint: s3.us-east-1.wasabisys.com
access key: abcdefg1234
secret key: abcdefg1234
set region ON
region: us-east-1
NOTE: If you are using s3.wasabisys.com for endpoint (which is us-east-1) you can skip the region setting here.
DigitalOcean:
Not the cheapest but some of you may be in the ecosystem so here you go.
bucketname: muhbucket
endpoint: nyc3.digitaloceanspaces.com
access key: abcdefg1234
secret key: abcdefg1234
set region ON
region: nyc3
iDrive:
Fairly new on the S3 front, as a company, around a LONG time. Very inexpensive.
Configure as follows: In your iDrive control panel, create a public bucket. Then click the gear icon on the bucket in the bucket lists. Get the PUBLIC BUCKET URL! - it will end with .dev - the part WITHOUT your bucket name is what you want
So if it is this:
https://muhbucket.w8q1.c19.e2-1.dev
Then when setting up the new storage setting this is what you use:
Bucket name: muhbucket
endpoint: w8q1.c19.e2-1.dev
access key : whatever
secret key: whatever
set region: OFF
With ALL of these services, please test. Your circumstances may differ and I offer no guarantees. Just covering my support posterior. Again, this a couple tweaks to allow attachment downloads mainly for alternative S3 providers. The core S3 code is still all IPS.
S3 Compatible Downloads Users: If you are happy with the old plugin, it still works - and remains working on many sites. The code here is a little bit better plus the Backblaze API downloads bypassing any S3 problems that might be encountered. The last version of S3CD was for 4.5 so if you are looking to get something more up-to-date, support the dev, and get some nifty support widgets, purchase this, make sure it is enabled, and then delete the old S3 Compatible Downloads plugin. That's it for upgrading. I should also note that I do not have any of the hacky http insecure link support here, and those of you with wacky bucket names need to get religion on that if you want to upgrade to this. I'll listen, but support on those fronts is gonna be a wait-and-see-maybe sort of thing. It's not 2015 anymore.
Status Widgets Heads-Up: These are scraping, parsing things - either the status pages themselves or the, becoming more rare by the day, rss/xml feeds. If any of these guys changes things up, the widgets will break (or rather the underlying data will). I'll be patching these things as they occur. This app has actually been running on my sites for a couple years now and I think I've had maybe one or two breaks in that time across all the widgets.
Invision Community 5: No idea at this stage what is and will not be possible here. Most of my applications will be new purchases for the new framework but THIS ONE, if it is upgradable to the 5.x line, will just get the 5.x version added in to the mix for this purchase.