compare complement test results to a baseline

One thing that might be neat in the future is noticing differing results
while the tests are still running, and modifying the log messages to
indicate them. I can imagine situations where you would want to abort
the test run immediately after seeing the first regression.

complement-baseline.tsv

@@ -0,0 +1,220 @@
+test	result
+GLOBAL	FAIL
+TestACLs	PASS
+TestBannedUserCannotSendJoin	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room/event_with_mismatched_state_key	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room/invite_event	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room/join_event	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room/leave_event	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room/non-state_membership_event	FAIL
+TestCannotSendKnockViaSendKnockInMSC3787Room/regular_event	FAIL
+TestCannotSendNonJoinViaSendJoinV1	FAIL
+TestCannotSendNonJoinViaSendJoinV1/event_with_mismatched_state_key	PASS
+TestCannotSendNonJoinViaSendJoinV1/invite_event	PASS
+TestCannotSendNonJoinViaSendJoinV1/knock_event	PASS
+TestCannotSendNonJoinViaSendJoinV1/leave_event	FAIL
+TestCannotSendNonJoinViaSendJoinV1/non-state_membership_event	PASS
+TestCannotSendNonJoinViaSendJoinV1/regular_event	FAIL
+TestCannotSendNonJoinViaSendJoinV2	FAIL
+TestCannotSendNonJoinViaSendJoinV2/event_with_mismatched_state_key	PASS
+TestCannotSendNonJoinViaSendJoinV2/invite_event	PASS
+TestCannotSendNonJoinViaSendJoinV2/knock_event	PASS
+TestCannotSendNonJoinViaSendJoinV2/leave_event	FAIL
+TestCannotSendNonJoinViaSendJoinV2/non-state_membership_event	PASS
+TestCannotSendNonJoinViaSendJoinV2/regular_event	FAIL
+TestCannotSendNonKnockViaSendKnock	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1/event_with_mismatched_state_key	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1/invite_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1/join_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1/knock_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1/non-state_membership_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV1/regular_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2/event_with_mismatched_state_key	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2/invite_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2/join_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2/knock_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2/non-state_membership_event	FAIL
+TestCannotSendNonLeaveViaSendLeaveV2/regular_event	FAIL
+TestClientSpacesSummary	FAIL
+TestClientSpacesSummary/max_depth	FAIL
+TestClientSpacesSummary/pagination	FAIL
+TestClientSpacesSummary/query_whole_graph	FAIL
+TestClientSpacesSummary/redact_link	FAIL
+TestClientSpacesSummary/suggested_only	FAIL
+TestClientSpacesSummaryJoinRules	FAIL
+TestContentMediaV1	PASS
+TestDeviceListsUpdateOverFederation	FAIL
+TestDeviceListsUpdateOverFederation/good_connectivity	FAIL
+TestDeviceListsUpdateOverFederation/interrupted_connectivity	FAIL
+TestDeviceListsUpdateOverFederation/stopped_server	FAIL
+TestDeviceListsUpdateOverFederationOnRoomJoin	FAIL
+TestEventAuth	FAIL
+TestFederatedClientSpaces	FAIL
+TestFederationKeyUploadQuery	FAIL
+TestFederationKeyUploadQuery/Can_claim_remote_one_time_key_using_POST	FAIL
+TestFederationKeyUploadQuery/Can_query_remote_device_keys_using_POST	FAIL
+TestFederationRedactSendsWithoutEvent	PASS
+TestFederationRejectInvite	FAIL
+TestFederationRoomsInvite	FAIL
+TestFederationRoomsInvite/Parallel	FAIL
+TestFederationRoomsInvite/Parallel/Invited_user_can_reject_invite_over_federation	FAIL
+TestFederationRoomsInvite/Parallel/Invited_user_can_reject_invite_over_federation_for_empty_room	PASS
+TestFederationRoomsInvite/Parallel/Invited_user_can_reject_invite_over_federation_several_times	FAIL
+TestFederationRoomsInvite/Parallel/Invited_user_has_'is_direct'_flag_in_prev_content_after_joining	PASS
+TestFederationRoomsInvite/Parallel/Remote_invited_user_can_see_room_metadata	PASS
+TestFederationThumbnail	PASS
+TestGetMissingEventsGapFilling	FAIL
+TestInboundCanReturnMissingEvents	FAIL
+TestInboundCanReturnMissingEvents/Inbound_federation_can_return_missing_events_for_invited_visibility	FAIL
+TestInboundCanReturnMissingEvents/Inbound_federation_can_return_missing_events_for_joined_visibility	FAIL
+TestInboundCanReturnMissingEvents/Inbound_federation_can_return_missing_events_for_shared_visibility	FAIL
+TestInboundCanReturnMissingEvents/Inbound_federation_can_return_missing_events_for_world_readable_visibility	FAIL
+TestInboundFederationKeys	PASS
+TestInboundFederationProfile	PASS
+TestInboundFederationProfile/Inbound_federation_can_query_profile_data	PASS
+TestInboundFederationProfile/Non-numeric_ports_in_server_names_are_rejected	PASS
+TestInboundFederationRejectsEventsWithRejectedAuthEvents	FAIL
+TestIsDirectFlagFederation	PASS
+TestIsDirectFlagLocal	PASS
+TestJoinFederatedRoomFailOver	PASS
+TestJoinFederatedRoomFromApplicationServiceBridgeUser	FAIL
+TestJoinFederatedRoomFromApplicationServiceBridgeUser/join_remote_federated_room_as_application_service_user	FAIL
+TestJoinFederatedRoomWithUnverifiableEvents	PASS
+TestJoinFederatedRoomWithUnverifiableEvents//send_join_response_missing_signatures_shouldn't_block_room_join	PASS
+TestJoinFederatedRoomWithUnverifiableEvents//send_join_response_with_bad_signatures_shouldn't_block_room_join	PASS
+TestJoinFederatedRoomWithUnverifiableEvents//send_join_response_with_state_with_unverifiable_auth_events_shouldn't_block_room_join	PASS
+TestJoinFederatedRoomWithUnverifiableEvents//send_join_response_with_unobtainable_keys_shouldn't_block_room_join	PASS
+TestJoinViaRoomIDAndServerName	PASS
+TestJumpToDateEndpoint	FAIL
+TestJumpToDateEndpoint/parallel	FAIL
+TestJumpToDateEndpoint/parallel/federation	FAIL
+TestJumpToDateEndpoint/parallel/federation/can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint	FAIL
+TestJumpToDateEndpoint/parallel/federation/looking_backwards,_should_be_able_to_find_event_that_was_sent_before_we_joined	FAIL
+TestJumpToDateEndpoint/parallel/federation/looking_forwards,_should_be_able_to_find_event_that_was_sent_before_we_joined	FAIL
+TestJumpToDateEndpoint/parallel/federation/when_looking_backwards_before_the_room_was_created,_should_be_able_to_find_event_that_was_imported	FAIL
+TestJumpToDateEndpoint/parallel/should_find_event_after_given_timestmap	FAIL
+TestJumpToDateEndpoint/parallel/should_find_event_before_given_timestmap	FAIL
+TestJumpToDateEndpoint/parallel/should_find_next_event_topologically_after_given_timestmap_when_all_message_timestamps_are_the_same	FAIL
+TestJumpToDateEndpoint/parallel/should_find_next_event_topologically_before_given_timestamp_when_all_message_timestamps_are_the_same	FAIL
+TestJumpToDateEndpoint/parallel/should_find_nothing_after_the_latest_timestmap	PASS
+TestJumpToDateEndpoint/parallel/should_find_nothing_before_the_earliest_timestmap	PASS
+TestJumpToDateEndpoint/parallel/should_not_be_able_to_query_a_private_room_you_are_not_a_member_of	FAIL
+TestJumpToDateEndpoint/parallel/should_not_be_able_to_query_a_public_room_you_are_not_a_member_of	FAIL
+TestKnockRoomsInPublicRoomsDirectory	FAIL
+TestKnockRoomsInPublicRoomsDirectoryInMSC3787Room	FAIL
+TestKnocking	FAIL
+TestKnockingInMSC3787Room	FAIL
+TestKnockingInMSC3787Room/A_user_can_knock_on_a_room_without_a_reason	FAIL
+TestKnockingInMSC3787Room/A_user_can_knock_on_a_room_without_a_reason#01	FAIL
+TestKnockingInMSC3787Room/A_user_cannot_knock_on_a_room_they_are_already_in	FAIL
+TestKnockingInMSC3787Room/A_user_cannot_knock_on_a_room_they_are_already_in#01	FAIL
+TestKnockingInMSC3787Room/A_user_cannot_knock_on_a_room_they_are_already_invited_to	FAIL
+TestKnockingInMSC3787Room/A_user_cannot_knock_on_a_room_they_are_already_invited_to#01	FAIL
+TestKnockingInMSC3787Room/A_user_in_the_room_can_accept_a_knock	PASS
+TestKnockingInMSC3787Room/A_user_in_the_room_can_accept_a_knock#01	PASS
+TestKnockingInMSC3787Room/A_user_in_the_room_can_reject_a_knock	FAIL
+TestKnockingInMSC3787Room/A_user_in_the_room_can_reject_a_knock#01	FAIL
+TestKnockingInMSC3787Room/A_user_that_has_already_knocked_is_allowed_to_knock_again_on_the_same_room	FAIL
+TestKnockingInMSC3787Room/A_user_that_has_already_knocked_is_allowed_to_knock_again_on_the_same_room#01	FAIL
+TestKnockingInMSC3787Room/A_user_that_has_knocked_on_a_local_room_can_rescind_their_knock_and_then_knock_again	FAIL
+TestKnockingInMSC3787Room/A_user_that_is_banned_from_a_room_cannot_knock_on_it	FAIL
+TestKnockingInMSC3787Room/A_user_that_is_banned_from_a_room_cannot_knock_on_it#01	FAIL
+TestKnockingInMSC3787Room/Attempting_to_join_a_room_with_join_rule_'knock'_without_an_invite_should_fail	FAIL
+TestKnockingInMSC3787Room/Attempting_to_join_a_room_with_join_rule_'knock'_without_an_invite_should_fail#01	FAIL
+TestKnockingInMSC3787Room/Change_the_join_rule_of_a_room_from_'invite'_to_'knock'	PASS
+TestKnockingInMSC3787Room/Change_the_join_rule_of_a_room_from_'invite'_to_'knock'#01	PASS
+TestKnockingInMSC3787Room/Knocking_on_a_room_with_a_join_rule_other_than_'knock'_should_fail	FAIL
+TestKnockingInMSC3787Room/Knocking_on_a_room_with_a_join_rule_other_than_'knock'_should_fail#01	FAIL
+TestKnockingInMSC3787Room/Knocking_on_a_room_with_join_rule_'knock'_should_succeed	FAIL
+TestKnockingInMSC3787Room/Knocking_on_a_room_with_join_rule_'knock'_should_succeed#01	FAIL
+TestKnockingInMSC3787Room/Users_in_the_room_see_a_user's_membership_update_when_they_knock	FAIL
+TestKnockingInMSC3787Room/Users_in_the_room_see_a_user's_membership_update_when_they_knock#01	FAIL
+TestLocalPngThumbnail	PASS
+TestLocalPngThumbnail/test_/_matrix/client/v1/media_endpoint	PASS
+TestMediaFilenames	FAIL
+TestMediaFilenames/Parallel	FAIL
+TestMediaFilenames/Parallel/ASCII	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_file_'ascii'	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_file_'ascii'_over_/_matrix/client/v1/media/download	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_file_'name;with;semicolons'	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_file_'name;with;semicolons'_over_/_matrix/client/v1/media/download	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_file_'name_with_spaces'	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_file_'name_with_spaces'_over_/_matrix/client/v1/media/download	FAIL
+TestMediaFilenames/Parallel/ASCII/Can_download_specifying_a_different_ASCII_file_name	PASS
+TestMediaFilenames/Parallel/ASCII/Can_download_specifying_a_different_ASCII_file_name_over__matrix/client/v1/media/download	PASS
+TestMediaFilenames/Parallel/ASCII/Can_upload_with_ASCII_file_name	PASS
+TestMediaFilenames/Parallel/Unicode	FAIL
+TestMediaFilenames/Parallel/Unicode/Can_download_specifying_a_different_Unicode_file_name	PASS
+TestMediaFilenames/Parallel/Unicode/Can_download_specifying_a_different_Unicode_file_name_over__matrix/client/v1/media/download	PASS
+TestMediaFilenames/Parallel/Unicode/Can_download_with_Unicode_file_name_locally	FAIL
+TestMediaFilenames/Parallel/Unicode/Can_download_with_Unicode_file_name_locally_over__matrix/client/v1/media/download	FAIL
+TestMediaFilenames/Parallel/Unicode/Can_download_with_Unicode_file_name_over_federation	FAIL
+TestMediaFilenames/Parallel/Unicode/Can_download_with_Unicode_file_name_over_federation_via__matrix/client/v1/media/download	FAIL
+TestMediaFilenames/Parallel/Unicode/Can_upload_with_Unicode_file_name	PASS
+TestMediaFilenames/Parallel/Unicode/Will_serve_safe_media_types_as_inline	SKIP
+TestMediaFilenames/Parallel/Unicode/Will_serve_safe_media_types_as_inline_via__matrix/client/v1/media/download	SKIP
+TestMediaFilenames/Parallel/Unicode/Will_serve_safe_media_types_with_parameters_as_inline	SKIP
+TestMediaFilenames/Parallel/Unicode/Will_serve_safe_media_types_with_parameters_as_inline_via__matrix/client/v1/media/download	SKIP
+TestMediaFilenames/Parallel/Unicode/Will_serve_unsafe_media_types_as_attachments	SKIP
+TestMediaFilenames/Parallel/Unicode/Will_serve_unsafe_media_types_as_attachments_via__matrix/client/v1/media/download	SKIP
+TestMediaWithoutFileName	FAIL
+TestMediaWithoutFileName/parallel	FAIL
+TestMediaWithoutFileName/parallel/Can_download_without_a_file_name_locally	PASS
+TestMediaWithoutFileName/parallel/Can_download_without_a_file_name_over_federation	FAIL
+TestMediaWithoutFileName/parallel/Can_upload_without_a_file_name	PASS
+TestMediaWithoutFileNameCSMediaV1	PASS
+TestMediaWithoutFileNameCSMediaV1/parallel	PASS
+TestMediaWithoutFileNameCSMediaV1/parallel/Can_download_without_a_file_name_locally	PASS
+TestMediaWithoutFileNameCSMediaV1/parallel/Can_download_without_a_file_name_over_federation	PASS
+TestMediaWithoutFileNameCSMediaV1/parallel/Can_upload_without_a_file_name	PASS
+TestNetworkPartitionOrdering	FAIL
+TestOutboundFederationIgnoresMissingEventWithBadJSONForRoomVersion6	FAIL
+TestOutboundFederationProfile	PASS
+TestOutboundFederationProfile/Outbound_federation_can_query_profile_data	PASS
+TestOutboundFederationSend	PASS
+TestRemoteAliasRequestsUnderstandUnicode	PASS
+TestRemotePngThumbnail	PASS
+TestRemotePngThumbnail/test_/_matrix/client/v1/media_endpoint	PASS
+TestRemotePresence	FAIL
+TestRemotePresence/Presence_changes_are_also_reported_to_remote_room_members	FAIL
+TestRemotePresence/Presence_changes_to_UNAVAILABLE_are_reported_to_remote_room_members	FAIL
+TestRemoteTyping	FAIL
+TestRestrictedRoomsLocalJoin	FAIL
+TestRestrictedRoomsLocalJoinInMSC3787Room	FAIL
+TestRestrictedRoomsLocalJoinInMSC3787Room/Join_should_fail_initially	PASS
+TestRestrictedRoomsLocalJoinInMSC3787Room/Join_should_fail_when_left_allowed_room	FAIL
+TestRestrictedRoomsLocalJoinInMSC3787Room/Join_should_fail_with_mangled_join_rules	PASS
+TestRestrictedRoomsLocalJoinInMSC3787Room/Join_should_succeed_when_invited	FAIL
+TestRestrictedRoomsLocalJoinInMSC3787Room/Join_should_succeed_when_joined_to_allowed_room	PASS
+TestRestrictedRoomsRemoteJoin	FAIL
+TestRestrictedRoomsRemoteJoinFailOver	FAIL
+TestRestrictedRoomsRemoteJoinFailOverInMSC3787Room	FAIL
+TestRestrictedRoomsRemoteJoinInMSC3787Room	FAIL
+TestRestrictedRoomsRemoteJoinInMSC3787Room/Join_should_fail_initially	PASS
+TestRestrictedRoomsRemoteJoinInMSC3787Room/Join_should_fail_when_left_allowed_room	PASS
+TestRestrictedRoomsRemoteJoinInMSC3787Room/Join_should_fail_with_mangled_join_rules	PASS
+TestRestrictedRoomsRemoteJoinInMSC3787Room/Join_should_succeed_when_invited	PASS
+TestRestrictedRoomsRemoteJoinInMSC3787Room/Join_should_succeed_when_joined_to_allowed_room	FAIL
+TestRestrictedRoomsRemoteJoinLocalUser	FAIL
+TestRestrictedRoomsRemoteJoinLocalUserInMSC3787Room	FAIL
+TestRestrictedRoomsSpacesSummaryFederation	FAIL
+TestRestrictedRoomsSpacesSummaryLocal	FAIL
+TestSendJoinPartialStateResponse	SKIP
+TestSyncOmitsStateChangeOnFilteredEvents	PASS
+TestToDeviceMessagesOverFederation	FAIL
+TestToDeviceMessagesOverFederation/good_connectivity	PASS
+TestToDeviceMessagesOverFederation/interrupted_connectivity	FAIL
+TestToDeviceMessagesOverFederation/stopped_server	FAIL
+TestUnbanViaInvite	FAIL
+TestUnknownEndpoints	FAIL
+TestUnknownEndpoints/Client-server_endpoints	PASS
+TestUnknownEndpoints/Key_endpoints	FAIL
+TestUnknownEndpoints/Media_endpoints	PASS
+TestUnknownEndpoints/Server-server_endpoints	PASS
+TestUnknownEndpoints/Unknown_prefix	PASS
+TestUnrejectRejectedEvents	FAIL
+TestUserAppearsInChangedDeviceListOnJoinOverFederation	PASS
+TestWriteMDirectAccountData	PASS