This was based on the similar section in the hedgedoc. I dropped the bit
about grapevine users getting banned from the conduwuit rooms cause we
haven't heard about that happening for a while and the original case was
pretty unclear.
Previously attempting to delete an MXC that is only associated with
dangling thumbnails would fail, because it assumes that every thumbnail
must have a corresponding original in the db, and errors out if it can't
find the original. This is incorrect because we create dangling
thumbnails when requesting a remote thumbnail over federation when we
don't have the original file.
When requesting remote thumbnails over federation, we can end up with a
thumbnail in the media db without an associated original file. Because
of this, skipping thumbnails is insufficient to get a list of all MXCs.
Previously we were only skipping thumbnails that had both dimensions
nonzero. I think the assumption was that only the dimensions allowed by
services::media::thumbnail_properties would be used. This is not the
case because we have used arbitrary thumbnail dimensions when requesting
remote thumbnails.
Now that we are able to distinguish between corrupted media keys and
missing files, it makes more sense to propagate the corrupted keys up to
the caller.
This is useful to easily distinguish missing files from corrupted keys.
All existing usage sites have been modified so there is no behavior
change in this commit.
Ruma dropped a couple dependencies and includes a stateres performance
improvement. May as well pull in everything else (except OTel) while
we're at it.
We *should* ensure that media deletion is always successful, but when a
bug causes a single object to fail deletion it's better to try to delete
the remaining objects than to give up entirely.
Our current code should never write these, because we have an allowlist
of thumbnail sizes. None of the allowed sizes contain a 0xFF byte. We
have observed keys with a 0xFF in the thumbnail size a couple times on
real servers, and believe an early version of conduit wrote these before
the allowlist was added.
These keys were originally handled correctly, and were broken by
e2cba15ed2. Before that commit, we were
parsing media keys backwards, and never tried to read the thumbnail size
or mxc url.
This became a problem because #foundation-office:matrix.org has a
malformed create event with its `predecessor` set to a string instead of
a map.
The solution to this is, unfortunately, to do more shotgun parsing to
extract only the desired fields rather than trying to parse the entire
content every time. To prevent this kind of problem from happening
again, `RoomCreateEventContent` must only be used for creating new PDUs,
existing PDUs must be shotgun-parsed.
This gets rid of 3 instances of re-parsing the room version.
There's one place where we need the event ID of the room create event to
verify federation responses, so now we just look up the event ID at that
point instead.
And some supporting changes:
* crane: It removed its dependency on nixpkgs and made overrideToolchain
take a function for splicing reasons, but we're doing splicing
ourselves so we can just ignore the function argument. These changes
are in `flake.nix`.
* [NixOS/nixpkgs#347228][0]: linkerFor* were removed because the linker
no longer needs to be different in some edge cases. Based on the
diff of the PR that introduced this change, ccFor* are the proper
replacements. These changes are in `cross-compilation-env.nix` in the
compiler-and-linker-choosing section.
* [NixOS/nixpkgs#350299][1]: buildPlatform isn't at the top level
anymore, we have to go through stdenv now. These changes are in
`nix/shell.nix`.
* rocksdb: nixpkgs has 9.6.1 now so we need to upgrade our rust
library to use the matching version. These changes are in
`Cargo.toml`, `Cargo.lock`, `nix/pkgs/default/default.nix`, and
`cross-compilation-env.nix` in the linker flags section.
[0]: https://github.com/NixOS/nixpkgs/pull/347228
[1]: https://github.com/NixOS/nixpkgs/pull/350299
Flake lock file updates:
• Updated input 'attic':
'github:zhaofengli/attic/4dbdbee45728d8ce5788db6461aaaa89d98081f0' (2024-03-29)
→ 'github:zhaofengli/attic/48c8b395bfbc6b76c7eae74df6c74351255a095c' (2024-10-30)
• Updated input 'attic/crane':
'github:ipetkov/crane/7195c00c272fdd92fc74e7d5a0a2844b9fadb2fb' (2023-12-18)
→ 'github:ipetkov/crane/4c6c77920b8d44cd6660c1621dea6b3fc4b4c4f4' (2024-08-06)
• Updated input 'attic/flake-compat':
'github:edolstra/flake-compat/35bb57c0c8d8b62bbfd284272c928ceb64ddbde9' (2023-01-17)
→ 'github:edolstra/flake-compat/0f9255e01c2351cc7d116c072cb317785dd33b33' (2023-10-04)
• Added input 'attic/flake-parts':
'github:hercules-ci/flake-parts/8471fe90ad337a8074e957b69ca4d0089218391d' (2024-08-01)
• Added input 'attic/flake-parts/nixpkgs-lib':
follows 'attic/nixpkgs'
• Removed input 'attic/flake-utils'
• Updated input 'attic/nixpkgs':
'github:NixOS/nixpkgs/07262b18b97000d16a4bdb003418bd2fb067a932' (2024-03-25)
→ 'github:NixOS/nixpkgs/159be5db480d1df880a0135ca0bfed84c2f88353' (2024-09-11)
• Updated input 'attic/nixpkgs-stable':
'github:NixOS/nixpkgs/44733514b72e732bd49f5511bd0203dea9b9a434' (2024-03-26)
→ 'github:NixOS/nixpkgs/797f7dc49e0bc7fab4b57c021cdf68f595e47841' (2024-08-22)
• Added input 'attic/nix-github-actions':
'github:nix-community/nix-github-actions/e04df33f62cdcf93d73e9a04142464753a16db67' (2024-10-24)
• Added input 'attic/nix-github-actions/nixpkgs':
follows 'attic/nixpkgs'
• Updated input 'crane':
'github:ipetkov/crane/109987da061a1bf452f435f1653c47511587d919' (2024-05-24)
→ 'github:ipetkov/crane/498d9f122c413ee1154e8131ace5a35a80d8fa76' (2024-10-27)
• Removed input 'crane/nixpkgs'
• Updated input 'fenix':
'github:nix-community/fenix/b6fc5035b28e36a98370d0eac44f4ef3fd323df6' (2024-05-22)
→ 'github:nix-community/fenix/87b4d20f896c99018dde4702a9c6157b516f2a76' (2024-11-01)
• Updated input 'fenix/rust-analyzer-src':
'github:rust-lang/rust-analyzer/21ec8f523812b88418b2bfc64240c62b3dd967bd' (2024-05-19)
→ 'github:rust-lang/rust-analyzer/0ba893e1a00d92557ac91efb771d72eee36ca687' (2024-10-31)
• Updated input 'flake-utils':
'github:numtide/flake-utils/b1d9ab70662946ef0850d488da1c9019f3a9752a' (2024-03-11)
→ 'github:numtide/flake-utils/c1dfcf08411b08f6b8615f7d8971a2bfa81d5e8a' (2024-09-17)
• Updated input 'nix-filter':
'github:numtide/nix-filter/3342559a24e85fc164b295c3444e8a139924675b' (2024-03-11)
→ 'github:numtide/nix-filter/776e68c1d014c3adde193a18db9d738458cd2ba4' (2024-10-29)
• Updated input 'nixpkgs':
'github:NixOS/nixpkgs/5710852ba686cc1fd0d3b8e22b3117d43ba374c2' (2024-05-21)
→ 'github:NixOS/nixpkgs/807e9154dcb16384b1b765ebe9cd2bba2ac287fd' (2024-10-29)
