4 years ago by benja123
Itās bad that Backblaze did not do their due diligence while integrating with Facebook pixel, but the bigger problem is the tendency of third party integrations having default settings that are overly aggressive when it comes to access of user/site data.
This should be a warning to every developer, when you integrate with any third party - donāt just copy the default snippet of code they recommend, read the docs thoroughly and then test/monitor what is being sent back to the third party. And if you are writing a third party widget, be considerate and make the defaults the least aggressive possible when it comes to accessing user/site/server data.
4 years ago by beshrkayali
At this point, every developer should (and needs) to be aware of how invasive any tracking code they blindly copy/paste into a website. I think it's required from people who operate normal websites, blogs, etc... but missing it is somewhat forgivable, especially from people just starting.
But the level of irresponsibility with customers' most private data from a company who's MAIN JOB is to protect it is absolutely shocking. Yes, it's the freaking pixel that does the tracking, BUT IT'S THEIR RESPONSIBILITY TO KNOW WHAT THE HECK IS HAPPENING ON THEIR WEBSITE. Don't they have any sort of vulnerability assessment or security code review? Their reply tweet is almost as infuriating as how something like that could even happen to begin with. Like yeah, sorry, ain't our fault! It's astounding.
I considered using backblaze a couple of times and now I'm very happy that I eventually didn't.
4 years ago by csnover
> Don't they have any sort of vulnerability assessment or security code review?
I havenāt seen any evidence that they do. The last time I brought up their history of bad security practices on HN, one of their co-founders decided that the correct course of action was to come on here, accuse me of being a bad actor, and repeatedly make up quotes I didnāt say.[0] All because I tried to warn others in the community that something just like this was likely to happen again. And now it has. So, you know.
4 years ago by tidepod12
Wow. After reading about the FB tracking I was wary about Backblaze but teetering on the edge of willing to give it a pass if they fixed it. But after reading brianwski's comments in that thread, the arrogance and unprofessionalism (especially in his last comment) just completely turned me off. That attitude goes beyond a technical fuckup or bad marketing move. I'm moving my backups out of Backblaze today and won't be looking back.
4 years ago by beshrkayali
Astonishing arrogance. Their replies on current incident are also dismissive and arrogant. I canāt even imagine what kind of culture they have internally.
4 years ago by vesinisa
> Don't they have any sort of vulnerability assessment or security code review?
I bet people just don't realize that the frontend code could end up being a source of major data confidentiality vulnerability. The threat modeling, auditing etc. usually just concentrates on attack scenarios involving the backend to save money and keep frontend development a bit lighter on the security review process side.
Does not make it excusable of course, just means their threat modeling was inadequate. But probably explains how this was able to sip into production.
4 years ago by Aeolun
I dunno, you donāt need to be particularly concerned with security to understand that you do not need a facebook tracking pixel on your āpaying customerā UI.
4 years ago by hertzrat
> I considered using backblaze a couple of times and now I'm very happy that I eventually didn't.
Same. I ended up going with spideroak because it was too hard to figure out what other providers were even promising for privacy, and itās been fine. Iām thinking maybe of trying rsync.net with duplicity one day
4 years ago by magicalhippo
> Itās bad that Backblaze did not do their due diligence while integrating with Facebook pixel
Here's the thing though, being a third party, due diligence means you constantly have to check what they're doing.
Because otherwise how do you know when they suddenly change their script to do something entirely different?
This is why having any 3rd party scripts on a dashboard service like this is in my view entirely inexcusable.
When I go to visit my cloud-based dashboard for Acronis' backup service[1] there's one single domain involved, and that's how it should be.
4 years ago by the8472
> Because otherwise how do you know when they suddenly change their script to do something entirely different?
https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
4 years ago by magicalhippo
Not bad, but as far as I can see it requires you are _really_ sure no further scripts are dynamically loaded.
Or is there a way for the server to specify all resources must have SRI?
4 years ago by thereddaikon
Why would you allow third party scripts to be updated by anyone but you?
This trend of dynamically linking to other people's shit needs to stop.
4 years ago by dexterdog
If you're updating them then they're not 3rd party scripts.
4 years ago by barbazoo
> Because otherwise how do you know when they suddenly change their script to do something entirely different?
Even though it's inconvenient maybe we should treat it as just another 3rd party dependency that needs to be downloaded, screened, and then used from the internal store. Pretty dangerous to dynamically load a script from a site like facebook.com.
4 years ago by coldtea
>Because otherwise how do you know when they suddenly change their script to do something entirely different?
(a) Sandbox it.
(b) Have them sign a contract they won't do so but only do these N things, and if change those without telling you, sue them.
4 years ago by gruturo
As much as I loathe and despise Facebook, this one is on Backblaze: they integrated with a well-known evil (seriously this surprises absolutely nobody) and should have been more careful with their settings. Or, you know, not have done it at all.
4 years ago by tornato7
Why does Backblaze even need Facebook integration anyway? That seems absurd to me.
4 years ago by bambax
> but the bigger problem is the tendency of third party integrations having default settings
It's certainly a problem, but the biggest problem is simply that Backblaze doesn't need to integrate Facebook pixel to their web interface, especially when users are logged in.
There is absolutely zero benefit to this for their users, but I also fail to see what it could bring to them as a company??
They have now created a major PR problem for themselves, and what did they get in exchange?
(Ok, as the saying goes, there's no such thing as bad publicity. But still.)
4 years ago by rciorba
The FB pixel is how FB gets to know about conversions. If your company runs FB adds, FB encourages you to let them know about which users converted to paying customers so it can improve the targeting of your ad campaign. And if you offer a free trial and the upgrade to paying customer happens inside the user dashboard, presto-bingo Marketing now needs you to install the FB pixel there.
It's disappointing that this is what the internet has become, and I'm happy to see issues like this brought forward. I can see value in feeding back conversion events to an ad network, but the "just let us run code on your page" style of integration needs to stop. Give developers some API to explicitly send such an event, if they really need to.
4 years ago by jeffasinger
Almost every ad network does give you some type of API like what you're talking about, but unfortunately they usually actively discourage use of it.
4 years ago by cosmie
PSA for any devs out there implementing FB Pixels:
ā Facebook's pixel will, by default, attach click listeners to the page and send back associated metadata. This simplifies implementation needs, but can create unintentional information leaks in privileged contexts. To disable this behavior, there's a flag[1] you can use. After which, you can manually trigger FB Pixel hits and control both when they're fired and what information is included in them.
ā There's a feature for the FB Pixel called Advanced Matching[2] that allows you to send hashed PII as parameters with your FB events. "Automatic Advanced Matching" can be enabled at any time via a toggle in the FB interface. I believe that setting autoConfig to false as mentioned above will similarly prevent Automatic Advanced Matching from working (since it disables the auto-creation of all those listeners to begin with). When manually triggering pixel calls like above, you can use this functionality via "Manual Advanced Matching"[3].
As a general rule, I'd strongly encourage anyone implementing a Facebook pixel to also include the autoConfig = false flag. This makes it work like most other pixels, where the base tag just instantiates an object. After which, hits only occur when explicitly defined in the site code, and include specifically what details you include in it. That way you're fully aware of the scope of data disclosure happening and any need from marketing to include sensitive (or potentially sensitive) information in these calls has to be explicitly requested (and theoretically vetted) as part of the standard dev process.
[1] https://developers.facebook.com/docs/facebook-pixel/advanced...
[2] https://www.facebook.com/business/help/611774685654668
[3] https://developers.facebook.com/docs/facebook-pixel/advanced...
4 years ago by miked85
This is good info, but Backblaze shouldn't even be using FB pixels to begin with.
4 years ago by cosmie
I don't necessarily disagree with you, and whether a pixel should be there at all is definitely a discussion in itself.
But for those who are implementing FB Pixels, I wanted to put out some potentially useful information that can help protect against unintended data disclosure, after mentioning the auto-listener behavior in a reply to another comment and being met with surprise[1].
4 years ago by nerdponx
Seems like it's designed to make it easy to accidentally send more of your users'/customers' data back to FB than intended. Oopsie!
4 years ago by KingOfCoders
Exactly. When the most important thing for a backup provider is the trust people put in you, don't do anything that risks that trust.
4 years ago by gigatexal
Edit: my biases against Facebook keep me from making cogent points.
4 years ago by mvzvm
This feels like an almost intentional misunderstanding.
4 years ago by mvzvm
Why is that? Perhaps they get business value out of it?
4 years ago by croon
If that business value destroys trust in their main business, that's a problem.
4 years ago by tga
It looks like theyāre about to lose a bunch of business value because of it.
4 years ago by bschwindHN
Won't somebody please think of the business value???
4 years ago by tomaskafka
I assume they get business value by retargeting site visitors - to do this, just run the FB pixel (properly configured!) on a marketing pages of the website. Not in the logged-in part!
4 years ago by Corrado
The thing that concerns me about the FB Pixel (and GTM) is that the host is completely free to do anything and everything to the page. Even if they don't do anything "evil" today, tomorrow is a different story completely. This scares the pants off of me and makes me want to rip out any "tracking" that I've ever installed on any site anywhere. Actually, that's probably not a bad idea.
Are there no browser level protections for this type of thing? I thought CORS was supposed to prevent these activities from happening.
4 years ago by cosmie
Virtually all tracking boils down to 1x1 sized images getting embedded on the page, with various metadata being attached in that image call. The javascript libraries may include other functionality (like additional fingerprinting and such), but are primarily just convenient abstractions that generate and embed the the tracking images for you. Most provide the details needed[1] to build your own generator function, which would allow you to integrate the tracking you want while reducing your security exposure to third party code.
As for GTM ā a deployed container is self-contained. If you don't want to expose your site to third party code, but want to use GTM as a convenient control plane for configuration of tags and tagging rules, you can do that. Instead of using the standard snippet that loads the container from Google, you can just grab the generated javascript file for the container after a new deploy and self-host it. It gives you the convenience of GTM (central control plane for tagging-related stuff, versioning and commenting, etc) but without the security exposure of embedding externally hosted scripts.
[1] https://developers.facebook.com/docs/facebook-pixel/advanced...
4 years ago by tga
The actual 1x1 pixel is a leftover from the previous generation tracking tools, and even the page you liked to recommends _against_ using that method because it canāt spy on users enough.
Here we are talking about a tracking _script_ embedded in the page and sending to Facebook everything the user does (āstandard or custom events triggered by UI interactionsā).
Using only a pixel to track how users move around the app wouldnāt have landed Backblaze in as much hot water. Instead, it looks like the Facebook _tracking script_ (automatically) exfiltrated sensitive data like file names, and that crosses a limit.
4 years ago by sbarre
As a first-party site owner, subresource integrity checks[0] (that someone else already linked elsewhere in this thread) lets you at least determine, at the browser request level, if a third-party script has changed since you installed (and hopefully audited) it.
0: https://developer.mozilla.org/en-US/docs/Web/Security/Subres...
4 years ago by mhils
The ad-tech response to SRI is to provide script tags where the entrypoint script loads additional scripts, which you cannot pin. ĀÆ\_(ć)_/ĀÆ
4 years ago by overscore
For various reasons including this, advertising tracking is moving server-side, where the company can much more tightly control what gets sent to the vendors, and where third party JavaScript no longer has access to the DOM, network requests, or cookies.
4 years ago by londons_explore
And pesky users can't point out GDPR violations using the browser Developer Tools.
Server side analytics will prove much more powerful and opaque when it gets integrated deep enough into web dev stacks to work properly.
4 years ago by altano
CORS is about safely allowing two cooperating, different origins to communicate. In this case, Facebook and the host are cooperating.
4 years ago by Lukas_Skywalker
As a protection for the users, addons like Facebook Container for Firefox [0] can isolate all Facebook tracking and prevent the scripts from running on pages that are not facebook.com.
[0] https://addons.mozilla.org/en-US/firefox/addon/facebook-cont...
4 years ago by rciorba
Even just using an ad-blocker will prevent this: https://github.com/gorhill/uBlock
4 years ago by corobo
And if that doesn't tick your creepy boxes lets try the financials. If a user hits your tracking pixel they (and those like them) will more likely see ads similar to yours, meaning potential customers will be more expensive to obtain now.
Don't give data to Facebook lmao.
4 years ago by azernik
One easy way to avoid this kind of mistake in your own product: make a clear distinction between your publicly-facing web site ("corpweb") and your web app for logged-in users. Preferably, they should be served from separate infrastructure.
Corpweb should be as static as possible, except for whatever third-party JS the marketing professionals think is necessary. It's their job, they know what's best.
Your app should have zero third-party JS except for technical analytics (New Relic, Datadog, whatever).
(This distinction can be fuzzier for free services, and for consumer stuff with non-sensitive data; Backblaze is neither of those.)
4 years ago by dylan604
>It's their job, they know what's best.
That made me LOL. You'd like to think they know best. Most of the time, they do things because other people do things, but don't truly understand what the true ramifications are of their requests.
Marketing: "What's the big deal? All you have to do is add the 2 or 3 lines of JS to the site."
Devs: "Do you know exactly what that will do?"
Marketing: "It'll give us all sorts of useful metrics for free"
Devs: "Do you know if it is secure or will cause our site to become less secure due to vulns in the included JS? Will it cause the site's performance to become sluggish where we will get blamed? Do you know exactly what data is being collected, and will it affect any of our other obligations of maintaining this site?"
Marketing: "Um..., that's your job. We just want the data"
4 years ago by azernik
I'm speaking from the perspective of a site, like Backblaze, where web app and the site fulfill two separate functions. There are lots of cool metrics that marketing wants; any code they put into www.backblaze.com is pretty low cost, and usually done by a separate team than product.
The product site itself (usually app.example.com, but Backblaze seems to use secure.backblaze.com) actually contains customer data in the browser context, is under much higher base resource loads from its core functionality, and is used repeatedly in workflows where poor performance is painful to the user.
No one gives a shit if your pricing page takes 500ms to load instead of 100ms, or if a dozen social media companies who already know where you work learn what kinds of professional products you're looking for.
They do care if a file listing takes longer, a recipe opens slower, if word frequencies in confidential data are leaked to the world.
In most paid products, the non-paid part of a site has radically different performance and security requirements from the paid part, and forcing one to be built to the requirements of the other (in either direction) is wasteful, or dangerous, or both.
4 years ago by undefined
4 years ago by usrnm
> Most of the time, they do things because other people do things, but don't truly understand what the true ramifications are
To be fair, it's just as true for developers as it is for the marketing guys
4 years ago by capableweb
This is essentially true for every single human and the reason discussion forums has discussions, because we keep being unable to see the other persons perspective and think we need to spread "the truth" to them.
4 years ago by iamacyborg
Iām a marketer, canāt argue with the above.
4 years ago by cuu508
If you are doing business in the EU, then you have to be careful about using 3rd-party stuff (Google fonts, embedded Youtube videos, embedded maps, chat widgets, ads, pixels, analytics etc. etc.) on the public-facing website as well.
> It's their job, they know what's best.
That's like sending an alcoholic down the spirits aisle.
4 years ago by azernik
As long as they don't have the car keys, they can go knock themselves out.
Re: the more substantive legal points - there are off-the-shelf solutions (CMPs, for example) and easy checklists for complying in the setting of a static public-facing website. The web designers and brand managers I've worked with are more than capable of meeting clear industry standards.
It's inside a web app, where customer data is on the page and in JS scopes, where the product team is essential in safeguarding customer data.
4 years ago by g_p
> there are off-the-shelf solutions (CMPs, for example) and easy checklists for complying in the setting of a static public-facing website.
In my experience, the people implementing these often don't understand enough about the technology to be allowed to implement this. I wonder if "easy to use" tag managers are to blame, by allowing non-experts to add JS and other includes to webpages without process or scrutiny.
Check a few big brand-name websites, and look at whether they place (third party) cookies before the CMP has even been interacted with.
I can think of some major high street labels where the consent prompt is mere theatre.
4 years ago by cuu508
> there are off-the-shelf solutions (CMPs, for example) and easy checklists for complying in the setting of a static public-facing website.
Consent Management Platforms, things like Cookiebot?
In my experience, blocking 3rd-party HTTP requests, cookies, LocalStorage access etc. before consent is given ā is easy in simple cases, but can quickly get technical and tricky.
4 years ago by smolder
I agree with this approach but there's an issue with "it's their job, they know best". They will inevitably want to put the same tracking crap in the logged-in site as well, so they can see how visitors converted into users, how they used the site once they were users, how valuable that made them as customers, and so on. As long as FB/other analytics firm is saying "we can help you market better with additional data", marketers are going to advocate sharing it.
4 years ago by azernik
Which is why it has to be a two way street - when it comes to the product, specifically its security and performance, engineering needs to absolutely own the thing.
4 years ago by tarsinge
> Corpweb should be as static as possible, except for whatever third-party JS the marketing professionals think is necessary. It's their job, they know what's best.
I strongly disagree. Marketing professionals often lack technical understanding and are superficial about the consequences. This mentality is how you end up with engineers working for companies doing morally questionable choices because they just want to be a cog in the system instead of being concerned about the direction and how the company do business. Separations between services is a shared illusion, if something is against your values please tell your fellow human beings.
4 years ago by azernik
In a static website, where there are standard tools and checklists and web designers to walk them through it, marketing's lack of technical understanding is less of an issue. And in a B2B web app like Backblaze's and my own, the data exposed to the public web site is just not all that sensitive.
And I'm not talking as a prospective employee who "just wants to be a cog in the machine"; I'm talking as a founder and CTO who sets company goals. I'm worried about data leaks caused by poor implementation and short-sightedness, not those caused by company policy that I disagree with. If I disagree with company policy, I change it.
4 years ago by cuu508
I'd be interested to see a site where marketing professionals with limited technical understanding knocked themselves out, but they used standard tools and checklists, and it came out OK. Do you have any examples?
4 years ago by Zhenya
This is actually pretty crazy. You pay for this service, and then they share, what many consider to be pii, directly to facebook.
I'm guessing if you're logged into facebook, now FB can correlate:
1) you use a backup service
2) all the metadata from the file names
Whoops.
This is why my network runs pi-hole / diversion with all tracking blocked network wide.
4 years ago by tuwtuwtuwtuw
Who are the "many" that consider file name PII?
4 years ago by Nadya
sisters_graduation_2019.jpg
Rebecca_Range_will_and_final_testament.pdf
NadyaNayme - Resume - [Company Name].pdf
divorce_process.pdf
steps-to-take-after-a-car-accident.pdf
personal_injury_michael_perez_west_coast_trial_lawyers.pdf
embarassing_porno_filename.mp4
And this is a smalls subset of filenames that could not only provide PII but also potentially embarrassing or private information that isn't identifying but would be accompanied with files that are personally identifying.
4 years ago by Zhenya
Well said, nadya
4 years ago by stordoff
Just from looking at my own documents folder, "Stephen Tordoff - ESA Appeal SSCS1.pdf". That would reveal that I have a disability, and that I am (or have) claiming benefits for it. Not everyone would be happy with that being public knowledge, and I'd be less that thrilled if things like it were shared with Facebook for no reason.
4 years ago by ipnon
URLs, of which file names could be considered a subset, are definitely considered PII.[0]
4 years ago by thefunnyman
Iāve worked for multiple FAANGs and can assure you that all have considered file names to be PII. Pretty much anything that contains user input should be treated as PII.
4 years ago by berniemadoff69
Pay careful attention to the response from Backblaze:
> "Hi Brett! The pixels we use are primarily for audience building when we advertise on other platforms like Facebook for example." [...]
The carefully calculated cutesy "Hi Brett!" with the exclamation point is the same reason big tech companies use infantile graphics [0]: by seeming playful, they create the illusion they are a Safe Friend you can Trust.
4 years ago by Edman274
Using a salutation, and addressing someone by name is not a conspiracy to make people trust you. The things that you should care about are the banality of evil, and that no one believes that they've done anything wrong. I live in the Midwest and my job is to make low-impact CRUD applications for a small car insurance company. I would use the same salutation, because I have been taught that that is what I'm supposed to do. I wasn't coached in some session on how to trick people into thinking I'm their buddy - it just becomes part of the shared social vocabulary.
4 years ago by Scene_Cast2
Modern business vocabulary has shifted from "Dear Mr. LastName," to "Hi FirstName!". This shift happened first in more "trendy" places, although most everyone has already moved on to using informal language in customer relations.
I do agree with your point about banality of evil.
4 years ago by pbourke
I refuse, with every last fibre of my being, that fucking exclamation mark.
4 years ago by berniemadoff69
Backblaze is role-playing 'trusted friend' on twitter the same way McDonalds and Wendys get into 'fights' on twitter. It's just corporate playbook stuff; I wouldn't say it's a conspiracy. Here's the latest posts on backblaze's twitter: https://i.imgur.com/mMkylym.png
4 years ago by sneak
A corporate playbook is an agreed-upon-in-advance set of rules to follow by a group of people.
That's pretty much the definition of a conspiracy.
4 years ago by brianjunyinchan
Absolutely. Better to focus on what actually matters, and keep the action dense.
4 years ago by rPlayer6554
Dear Mr. Madoff69,
I believe that your analysis has flaws. It would be quite awkward on social media to use the a more formal way of addressing people. Twitter and other platforms have a "style" of conversation and trying to fit the square peg of formal writing into the round hole of internet conversation sounds stilted. I do not understand why you think corporations would decide to do that, no matter what their intentions are.
I humbly await your response.
Yours Truly, Mr. rPlayer
P.S. I hope your Grandmother is doing well. Please send my regards.
P.P.S. Please invest in my new cloud computing blockchain biotech startup where we sell NFTs.
4 years ago by TeMPOraL
Dear Mr. rPlayer,
Please update your signature to conform with the current standards, as outlined in the last month's circular.
Best Regards, TeMPOraL
--
TeMPOraL, Internet Compliance Officer (ICO)
ACME LLC - Synergizing Creative Accounting
ACME LLC, NaN NaN, Null Islands.
The content of this message is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.
Please do not print this message unless it is necessary. Every unprinted message helps the environment. Think of the trees!
4 years ago by dgellow
It's even better with the small tree icon hosted by an external service, thus leaking some data when you open the email :)
4 years ago by simias
I don't use twitter so I don't know what the etiquette over there but outside of emails I wouldn't normally expect any salutation in an internet message.
So for me it's not so much that the salutation not formal enough, it's more that it's odd that it exists at all.
But then again maybe usages differ in twitterworld.
4 years ago by ylyn
They could have just left off the greeting.
4 years ago by _etyf
The flat corporate graphical style has a Know Your Meme page and rich parody-art ecosystem:
https://knowyourmeme.com/memes/subcultures/corporate-art-sty...
4 years ago by exikyut
What nobody seems to be pointing out here: Ben's account contains exactly what you want to hand off to Someone Else's Storageā¢.
It's just GPG-encrypted backups.
Look. Here's what the innerText says, with the `\n`s interpreted:
Upload
Download
New Folder
Delete
Snapshot
Selected: 0 Files: 0 bytes
Name
Size
Uploaded
dbbackup_06_23_2017.sql.gpg
10.2 GB
06/23/2017 22:24
dbbackup_06_24_2017.sql.gpg
10.2 GB
06/24/2017 22:24
dbbackup_07_02_2017.sql.gpg
10.2 GB
07/02/2017 22:25
dbbackup_07_03_2017.sql.gpg
10.2 GB
07/03/2017 22:26
dbbackup_07_04_2017.sql.gpg
10.2 GB
07/04/2017 22:25
dbbackup_07_05_2017.sql.gpg
10.2 GB
07/05/2017 22:25
key.asc
3.9 KB
02/24/2016 22:01
(For the curious: I couldn't be bothered to install https://projectnaptha.com/ (probably would've worked), I just resized a terminal to the same column width and blindly retyped the text into a shell printf statement.)
4 years ago by cm2187
And also when you upload something to blackblaze, you are already handing out your files to a third party. There shouldn't be anything sensitive that you didn't encrypt client side.
4 years ago by nvarsj
Exactly... all my b2 files are client side encrypted. It would be a really bad idea not to do that.
4 years ago by dbmnt
Except... are the filenames encrypted? For many, if not most, filenames are plaintext. Backblaze's default "SSE-B2" encryption leaves the filenames in plaintext. GPG encryption typically leaves the filenames in plaintext. There's still information people consider private and didn't expect to be shared with Facebook.
4 years ago by aasasd
There are programs that automatically encrypt files before storing them. A particularly enticing one is Cryptomator, which acts as a ādiskā keeping files on various providers and thus syncs between devicesāhowever its Android app isn't open-source so I never actually tried it.
4 years ago by durnygbur
It's still revealing two levels of filename extesions - type of encryption and file format. Full filename should be simply random alphanumeric ASCII string without any filename extensions, although this requires to manage and store a map of keys and filenames.
4 years ago by exikyut
Hmm. A possible counterargument: filenames like
dRDDrOu44Rr84vzXJv2mcr2eg83zDN43mzUQ0N4
xzrI5Ha7HJ7gK3T8XfkGqNtvc7LMQPFjSwi
5Wb1XhHeR6LxQUC8XfyX9kvvooYvrp9fnxQVvH9C
jgAzjd56DGWFjcae0gw9A1LZxJEqVHW7UmkZ
XrpNNAZalPp6D4mnpLvVcCE3uWkDQzthSQwK9
are extremely suspicious-looking. To me that screams "obsessively paranoid". I reckon that if I were in law enforcement, the fact that there is absolutely nothing I can infer from these filenames, combined with the obvious complexity associated with correctly maintaining something like this, would actually make me that much more interested in decrypting this information simply to take a look at it and rule it out.
Which is exactly why this would be a scenario in which I _would_ want to "reuse someone else's password", if you will, and I'd theoretically go digging for common archival file patterns, and use the most common I came across.
4 years ago by DVk6dqsfyx5i3ii
> as generated by for example `head -c 30 /dev/urandom | base64 | tr -d '+/='`
If you want exactly 30 alphanumeric characters you could do `tr -dc '[:alnum:]' < /dev/urandom | head -c 30`
4 years ago by quchen
On-disk filenames should be hashes, much like .git contains lots of hash-named files. My Restic backup folders donāt show much beyond that they are Resic backup folders (if even?).
4 years ago by thebean11
Rclone crypt backends generate filenames like this, and are really not very difficult to set up.
4 years ago by KptMarchewa
>I reckon that if I were in law enforcement, the fact that there is absolutely nothing I can infer from these filenames, combined with the obvious complexity associated with correctly maintaining something like this, would actually make me that much more interested in decrypting this information simply to take a look at it and rule it out.
If I ever did something shady, I'd absolutely make a ton of honeypots like this.
4 years ago by guywhocodes
Yikes, that's a lot of good will burned. Doesn't matter if it is unintentional or not.
4 years ago by celsoazevedo
The inclusion of Facebook tracking was intentional:
https://twitter.com/backblaze/status/1373751015594356739
Not sure if they intended to send file names and sizes to FB, but in any case this doesn't look good. I'm currently looking for alternatives.
4 years ago by chunkles
rsync.net was posted here on HN a few days ago. Also, tarsnap is another popular service. Neither have the special additions that makes Backblaze so popular, but could be popular alternatives.
4 years ago by thayne
From a quick look at pricing pages it looks like rsync.net is 5x as expensive as backblaze b2, and has a minimum of 400gb per month (it also looks like you might have to preallocate vs pay on demand). And tarsnap is 10x as expensive as rsync.net.
My guess the bulk of that price difference is due to economies of scale.
4 years ago by gpm
So, looking at some potential things to switch to...
OVH has two interesting products here:
OVH cloud archive works with rsync, costs roughly half as much for storage as blackblaze, roughly the same for egress, but charges for ingress (at the same rate as egress).
OVH object store is s3 compatible (like blackblaze), charges roughly the same for bandwidth, 2x for storage.
https://www.ovhcloud.com/en/public-cloud/prices/#439
Digitalocean has a blob store with the same pricing on bandwidth, 4x the pricing on storage, a minimum spend of $5/month on storage, but the first tb ($10) of bandwidth free.
4 years ago by tbodt
The only reason I use Backblaze instead of tarsnap is it is 62 times cheaper for the same amount of storage. tarsnap is dramatically overpriced if you ask me.
4 years ago by Skunkleton
Rsync.net has a self-supported borg option that is a great deal.
4 years ago by BeefySwain
Can someone here translate the PR/marketing speak here for us mere mortals? How does having Facebook tracking on the web front-end of existing and paying users help with lead generation?
4 years ago by cosmie
Within Facebook, you can use the event stream collected by your FB Pixel to both define conversion criteria as well as create audiences and define inclusion/exclusion criteria for that audience. When it comes to tracking on pages behind auth, primarily it's for audience building which can be used for
ā Cross-sell/Up-sell campaigns. Build an audience based on usage patterns, and create a create a campaign for a complimentary service or higher tier (say, for example, someone clicks the button for a gated feature they don't have access to).
ā Suppression lists. If you don't want your campaigns to target existing users, you can build an audience from pixel data on your authenticated pages and suppress against that.
ā Lookalike audiences. After you create an audience in Facebook, you can create a "lookalike audience" from that. So even if you aren't actively doing either of the above, you'd derive value from tracking your "best" customers and using it as a seed list for a lookalike audience.
You're also not limited to using the FB Pixel for any of the above. In addition to a browser-side pixel, FB allows you to upload hashed customer information and use those for conversion tracking and audience building. Which used to be completely transparent to end users, but now you're able to see a list of companies that have uploaded your info to FB in this manner (I can't recall where it's buried in the user settings, off the top of my head).
All of that said, it's entirely likely that Backblaze wasn't intentionally sending any of this data to FB to begin with. An insidious aspect of FB's Pixel is that it automatically attaches listeners to a bunch of stuff on the page such as buttons and sends back interactions and associated metadata[1]. The flag to disable this isn't mentioned in the implementation instructions that are generated upfront, and it's actually a fairly uncommon trait for ad pixels. So a typical implementation tends to leave it on out of ignorance rather than make a deliberate determination on whether to use or disable that functionality.
[1] https://developers.facebook.com/docs/facebook-pixel/advanced...
4 years ago by yellow_postit
Assuming itās to build lookalike audiences [1] (find me people that act like the paying customers).
[1] https://www.facebook.com/business/help/164749007013531?id=40...
4 years ago by busymom0
That PR/marketing tweet comes across as from someone who doesn't understand how big deal this actually is and why customers won't be comfortable with a FB pixel on their dashboards with data filenames and sizes.
4 years ago by busymom0
Some might find this tweet useful for the time being:
> What happens if you resolve http://facebook.com to 127.0.0.1 via hosts-file? (Or put it into your Pi-Hole DNS Ad-Blocker, or the like.) Does the Backblaze UI still work?
> Answer: Seems to work well. You need to add the main domain and sub domains (www. in this case), btw.
4 years ago by teruakohatu
Including the filenames seems to have been unintentional, looks like they were logging analytics to Facebook, probably an even (form submission) but uploaded the form html with contents.
But why they need to submit that to Facebook for paying users I don't understand. The only thing I can think of is excluding active users from advertising... But is that worth the privacy intrusion?
4 years ago by robbiemitchell
They answered in the Twitter thread: they send data about paying users to Facebook so they can build lookalike audience targeting for new user acquisition. Other major ad platforms (Google, LinkedIn) have similar features.
This doesnāt look like the right data to be sending FB for that, though.
4 years ago by neilv
They could maybe salvage goodwill with genuine corporate soul-searching that ends up asserting/reasserting values -- and leads them to focus on providing trustworthy service to their users, and conspicuously away from some "tech" industry norms of selling out one's users.
As a provider of a paid service, it seems like they're in a better position to take the high road than a lot of tech companies are, but they have to decide that's who they are, and be clear they mean it.
4 years ago by kardos
Yep that pretty much burns the service :/
4 years ago by atYevP
Yev from Backblaze here -> weāve looked into and verified the issue and have pushed out a fix. We will continue to investigate and will provide updates as we have them.
4 years ago by mnw21cam
The exact phrasing you have used here is repeated multiple times in the Twitter thread, and I can only conclude that "pushed out a fix" is what your marketing department has decided to call what you have done.
What you have right here and right now is a public relations disaster. Trust in your brand has been damaged. It cannot be repaired by you providing minimal information. Your standardised message is akin to "Don't worry your little heads over the details - trust us, everything is fine now", and to be honest I find it a bit insulting. As far as we know, "pushed out a fix" could mean that you have hidden the tracking, so it is harder to find. Your short message is making the public relations disaster worse, not better.
These are the steps that you need to take:
1. Provide an explanation of why tracking was being performed in the first place, including an analysis of how much of that was a mistake.
2. Make an apology for breaching your customers' trust. This is a really important step, and it should be repeated in each of your press releases.
3. Provide details on the steps you have taken to fix the problem, and what that means for tracking data.
4. Make a promise that strictly limits the level of tracking that you will be allowing yourself to make in the future. Ideally we would all want that to be zero, and if you intend to do business with certain jurisdictions then you are limited to what is legal, but you must in any case be clear about what tracking you will ever do.
Honesty and transparency are the keys at this point to restoring your brand. I do not think the community will accept anything less.
4 years ago by elefantastisch
This.
Yev, if you are not in the upper management chain of Backblaze, please show mnw21cam's message to someone who is.
The problem is not that there was a little bug which caused the Facebook tracker to get a few little pieces of information it shouldn't have.
The problem is that Backblaze failed to understand how to distinguish appropriate and inappropriate uses of third-party trackers for signed-in users on a security-critical application. The Facebook pixel should never have been there at all. It shouldn't even have been considered. It should've been an absolute no-brainer that Facebook has no business being on secure pages on a critical infrastructure service for paying customers.
The fact that the pixel even showed up at all on a logged in page represents a breach of trust for customers and casts doubt on Backblaze's competence in handling security issues. This warrants a serious reply from the CEO, not a copy-pasted meaningless reassurance.
4 years ago by KindOne
OP is "Senior Director Of Marketing" according to Linkedin.
He posts a lot on HN and Reddit in Backblaze posts.
4 years ago by rsync
"What you have right here and right now is a public relations disaster. Trust in your brand has been damaged. It cannot be repaired by you providing minimal information."
I'm not sure that's true.
There are a number of barely-technical subreddits, typically centered around Plex or some flavor of bittorrent, that consist of an all-day-every-day request for "as much cloud storage as possible for the lowest possible cost and please say it's free".
These are not HN readers. It's as if they attained consciousness two minutes ago and one minute ago they decided they needed cloud storage.
This is the audience these kind of analytical tools are geared toward and I don't think this changes their "engagement" or their "convertibility" or their "lifetime value".
The real information here is that in 2021, and in conjunction with a much more sophisticated product offering (B2), Backblaze is very aggressively pursuing flat-rate, loss-leaders who can be influenced and targeted by facebook.
Draw what conclusions from that you will.
4 years ago by williamsmj
B2 is not flat rate pricing. It's $5/TB.
I understand that is a lower price than your own product, but it doesn't necessarily follow that they're losing money.
You seem to be implying that their covering the losses on storage by selling customer data. Is that correct? In which case, would you like to be a bit more specific and explicit?
4 years ago by leokennis
While on a personal level I agree with you, there is a saying in Dutch: "je moet niet wrijven in een vlek", which translates to "don't rub in a stain".
Sort of like the Streissand Effect; if BB starts posting press releases about "breaching trust", filled with apologies, the audience that will become aware of this issue will be a lot bigger than it currently is.
So from a PR persepctive, a one line low key "we pushed a fix" reply makes total sense.
4 years ago by dnzm
Being open and making clear that you understood you goofed would be total opposite of the Streisand effect - where you'd try to _silence_ discourse with the unintended effect of amplifying it.
Right now, the issue is being downplayed, much to the chagrin of an increasingly knowledgeable set of computer users.
4 years ago by atYevP
Yev here -> re: fixing the issue - we removed the offending code from the logged in web pages. Rest assured that we have are taking this very seriously internally and we will continue to investigate and provide updates as we have them.
4 years ago by tobr
Thanks for participating in this thread. As a longtime paying customer, I consider this a monumental security breach and I will be leaving the service. Itās clear that Backblaze have prioritized growth hacking or whatever over the security and privacy of me as a paying customer, and that your security processes are woefully inadequate.
4 years ago by skinkestek
Well, for many of us this is where it starts:
I might very well start using backblaze next year or maybe even next month[1], but that is depending on the outcome of this event.
For a comparison: one good friend once called med to apologize that he had been laughing behind my back with some friends.
Guess who I definitely trust today? The one admitted his mistake. He always was a nice bloke and I guess he will never ever do anything like that ever again.
[1]: I won't start using it this week or the next however.
4 years ago by tobr
What kind of scenario do you have in mind? I think itās possible to turn an incident like this around, PR-wise, but I canāt see how they will explain how they can sell something as secure and trusted, when their security process was unable to discover that they had deployed spyware in production. If it was there for a few hours before it was discovered and removed, well maybe.
4 years ago by ehnto
Just hope they announce the changes to security and implementation processes rather than just if they fix this issue or not. This really shouldn't have occurred in the first place, so you want to know they've fixed the root cause, bad process, not just the symptom.
4 years ago by atYevP
Yev here - Tobr, thank you for being a customer. Wanted to let you know that we've updated our blog post after finishing our root cause analysis. You can read about what happened here: https://www.backblaze.com/blog/privacy-update-third-party-tr....
4 years ago by Ensorceled
Hey Yev,
As another data point, Backblaze has pretty much until this weekend to provide an update that includes "we have removed Facebook and are getting a 3rd party review for other security holes in our product". After this weekend, I won't care because I'll be on a different product.
This is embarrassingly bad, as in, I'm now embarrassed for recommending using Backblaze at my company.
Crap, I just realized I got Backblaze installed at two previous companies. Thanks!
4 years ago by atYevP
Yev here -> thanks for being a customer! We've finished our root cause analysis and updated our blog post with additional information -> https://www.backblaze.com/blog/privacy-update-third-party-tr....
4 years ago by ohlookabird
Thanks. However, it is just not excusable and a breach of trust. The Backblaze Twitter communication making pretty clear that they don't see a problem with tracking paying customers. We are moving somewhere else.
4 years ago by dgellow
I wouldn't judge a company based on their Twitter engagement. Twitter is a really dumb place.
4 years ago by wccrawford
If a company engages via a public medium, you absolutely should judge them by their interactions on it.
If they refrain from engaging, then I agree that you shouldn't judge them by that.
4 years ago by atYevP
Yev here -> I think part of that was a mistake on the Twitter side of things. We've published a blog post with information after finishing our root cause analysis -> https://www.backblaze.com/blog/privacy-update-third-party-tr....
4 years ago by nerdponx
Same, I'm out. Who are you moving to?
4 years ago by quchen
Iāve looked at Backblaze and Wasabi for my personal off-site backups, and sided with Wasabi. Canāt complain so far.
4 years ago by ddalex
Is it smart to tell people that you are leaving before you actually backed-up and moved you data in the new place!?
4 years ago by tobr
I kinda lost whatever trust I had in Backblaze here, but I donāt think theyāre in the habit of deleting paying customersā data in vengeance.
Daily digest email
Get a daily email with the the top stories from Hacker News. No spam, unsubscribe at any time.