Skip to content

Commit 4e36fcc

Browse files
newrennewren
committed
send-pack: avoid sending the whole tree when pushing from a shallow clone
When pushing from a shallow clone, even if we only have made a small one-line change to a tiny file, we often push the entire toplevel tree of files. For large repositories, this could be gigabytes instead of kilobytes. The reason for this is that the push likely lacks the commits the receiver has advertised, so it walks back to its shallow grafts. Since it doesn't know that the server has anything, it sends the entire tree for the graft. It would also send the parents of the shallow graft, except the shallow clone doesn't have those by construction. We thus are forced to assume that the server has the parents of the shallow graft -- if it doesn't, the server's receive-pack will reject the push. But that raises the obvious question: if we're going to assume the server has the parents of the shallow graft, why not just assume the server has the shallow graft itself -- which this clone almost certainly received from the server when the shallow clone was created? As noted above, receive-pack already has a builtin connectivity check that predates pushing from a shallow clone by years[*], so even if a client is pushing to a different server than it cloned from, the worst that happens is a rejected push. And by assuming the server has the shallow graft commits, then for large repositories (those most likely to use shallow clone) we can avoid transferring (and perhaps re-compressing) gigabytes of file contents that the server already has. [*] Compare 5dbd767 (receive/send-pack: support pushing from a shallow clone, 2013-12-05) and 52fed6e (receive-pack: check connectivity before concluding "git push", 2011-09-02) Fix this by finding the shallow grafts behind the history we're pushing and adding them to the pack boundary as uninteresting (negative) tips, so the generated pack leaves out everything underneath them. We only use grafts that the pushed commits can actually reach; excluding every graft in the repository would be simpler, but it could drop an object we really do need to send -- for example, a new blob we're pushing that also happens to sit under some unrelated shallow root pulled from a different remote. We can also stop early at any commit we and the server both have -- one the server advertised, or that push negotiation found in common. Such a commit already marks the edge of what we need to send, so there's no reason to keep walking down to a graft below it. For deeper clones the server usually has a commit close by, which keeps this walk short; we only reach a graft when we and the server share no history that we know about. One very rare (and non-default) workflow genuinely needs the larger push: seeding a receiver willing to adopt new shallow roots (receive.shallowUpdate; see 5dbd767 (receive/send-pack: support pushing from a shallow clone, 2013-12-05) and 0a1bc12 (receive-pack: allow pushes that update .git/shallow, 2013-12-05)). When the server sets receive.shallowUpdate, it is willing to accept pushes despite lacking ancestors of the pushed commits. But it expects us to send all tree objects so it can graft a new shallow root. For that case, add a sender-side config, push.shallowExcludeBoundary, defaulting to true (the optimization), while allowing users to set it to false to restore the previous behavior needed for that rare case. Update the existing shallow-seeding tests in t5538 to set push.shallowExcludeBoundary=false, since they exercise that receive.shallowUpdate path. Add tests for the optimized default and the opt-out, that a rejected ref does not cause an accepted ref to be over-excluded, and that a shallowUpdate receiver still rejects a rootless snapshot by default. Signed-off-by: Elijah Newren <newren@gmail.com>
1 parent 9665003 commit 4e36fcc

3 files changed

Lines changed: 265 additions & 3 deletions

File tree

‎Documentation/config/push.adoc‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -134,6 +134,18 @@ This will result in only b (a and c are cleared).
134134
rely solely on the server's ref advertisement to find commits
135135
in common.
136136
137+
`push.shallowExcludeBoundary`::
138+
When pushing from a shallow repository (see linkgit:git-clone[1]
139+
`--depth`), Git normally assumes that the receiving end already
140+
has the pushing repository's shallow grafts, and omits those
141+
objects from the generated pack rather than resending the full
142+
toplevel tree of those grafts. This is safe because the
143+
receiving end rejects a push that references objects it does not
144+
have. Set this to `false` to send those objects anyway; this is
145+
only needed for the highly unusual case of using a push to seed
146+
a receiver that adopts new shallow roots (i.e. a receiver that
147+
has explicitly set `receive.shallowUpdate`). Default is `true`.
148+
137149
`push.useBitmaps`::
138150
If set to `false`, disable use of bitmaps for `git push` even if
139151
`pack.useBitmaps` is `true`, without preventing other git operations

‎send-pack.c‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
#include "transport.h"
1515
#include "version.h"
1616
#include "oid-array.h"
17+
#include "oidset.h"
1718
#include "gpg-interface.h"
1819
#include "shallow.h"
1920
#include "parse-options.h"
@@ -55,6 +56,91 @@ static void append_negative_object(struct repository *r,
5556
oid_array_append(haves, oid);
5657
}
5758

59+
static int check_to_send_update(const struct ref *ref, const struct send_pack_args *args);
60+
61+
/*
62+
* Add the shallow grafts (nr_parent == -1), which are reachable from the
63+
* refs being pushed, to the pack boundary ("haves") as uninteresting
64+
* (negative) tips so the generated pack leaves out everything beneath them.
65+
*
66+
* Walk only from the pushed tips, and only until a graft: using a graft
67+
* that does not bound the pushed history could exclude an object we are
68+
* genuinely sending (if it is also reachable from that unrelated graft).
69+
* Stop early at any commit the peer already has, since it is a negative
70+
* the peer can use and the graft beneath it would be redundant.
71+
*/
72+
static void append_reachable_shallow_grafts(struct repository *r,
73+
const struct ref *refs,
74+
const struct oid_array *advertised,
75+
const struct oid_array *negotiated,
76+
const struct send_pack_args *args,
77+
struct oid_array *haves)
78+
{
79+
struct commit_list *pending = NULL;
80+
struct oidset seen = OIDSET_INIT;
81+
struct oidset known = OIDSET_INIT;
82+
const struct ref *ref;
83+
size_t i;
84+
85+
for (i = 0; i < advertised->nr; i++)
86+
oidset_insert(&known, &advertised->oid[i]);
87+
for (i = 0; i < negotiated->nr; i++)
88+
oidset_insert(&known, &negotiated->oid[i]);
89+
90+
/*
91+
* Record every commit the peer is known to have as a boundary for
92+
* the walk, and seed the walk from the tips we are actually sending.
93+
* The walk below does not begin until "known" is fully populated.
94+
*/
95+
for (ref = refs; ref; ref = ref->next) {
96+
struct commit *commit;
97+
98+
if (!is_null_oid(&ref->old_oid))
99+
oidset_insert(&known, &ref->old_oid);
100+
101+
if (is_null_oid(&ref->new_oid))
102+
continue;
103+
if (check_to_send_update(ref, args))
104+
continue;
105+
commit = lookup_commit_reference_gently(r, &ref->new_oid, 1);
106+
if (commit)
107+
commit_list_insert(commit, &pending);
108+
}
109+
110+
while (pending) {
111+
struct commit *commit = pop_commit(&pending);
112+
const struct object_id *oid = &commit->object.oid;
113+
struct commit_graft *graft;
114+
struct commit_list *parent;
115+
116+
if (oidset_insert(&seen, oid))
117+
continue;
118+
119+
/*
120+
* A commit the peer already has bounds the pushed history
121+
* with a negative it can use, so stop here rather than
122+
* descend to a graft that would only be redundant.
123+
*/
124+
if (oidset_contains(&known, oid) &&
125+
odb_has_object(r->objects, oid, 0))
126+
continue;
127+
128+
graft = lookup_commit_graft(r, oid);
129+
if (graft && graft->nr_parent == -1) {
130+
append_negative_object(r, haves, oid);
131+
continue;
132+
}
133+
134+
if (repo_parse_commit(r, commit))
135+
continue;
136+
for (parent = commit->parents; parent; parent = parent->next)
137+
commit_list_insert(parent->item, &pending);
138+
}
139+
140+
oidset_clear(&seen);
141+
oidset_clear(&known);
142+
}
143+
58144
/*
59145
* Make a pack stream and spit it out into file descriptor fd
60146
*/
@@ -88,6 +174,20 @@ static int pack_objects(struct repository *r,
88174
for (size_t i = 0; i < negotiated->nr; i++)
89175
append_negative_object(r, &opts.haves, &negotiated->oid[i]);
90176

177+
/*
178+
* When pushing from a shallow repository, avoid re-pushing the
179+
* entire toplevel tree.
180+
*/
181+
if (is_repository_shallow(r)) {
182+
int exclude_boundary = 1;
183+
repo_config_get_bool(r, "push.shallowexcludeboundary",
184+
&exclude_boundary);
185+
if (exclude_boundary)
186+
append_reachable_shallow_grafts(r, refs, advertised,
187+
negotiated, args,
188+
&opts.haves);
189+
}
190+
91191
while (refs) {
92192
if (!is_null_oid(&refs->old_oid))
93193
append_negative_object(r, &opts.haves, &refs->old_oid);

‎t/t5538-push-shallow.sh‎

Lines changed: 153 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,8 @@ EOF
6464
test_expect_success 'push from shallow clone, with grafted roots' '
6565
(
6666
cd shallow2 &&
67-
test_must_fail git push ../.git +main:refs/remotes/shallow2/main 2>err &&
67+
test_must_fail git -c push.shallowExcludeBoundary=false \
68+
push ../.git +main:refs/remotes/shallow2/main 2>err &&
6869
test_grep "shallow2/main.*shallow update not allowed" err
6970
) &&
7071
test_must_fail git rev-parse shallow2/main &&
@@ -75,7 +76,8 @@ test_expect_success 'add new shallow root with receive.updateshallow on' '
7576
test_config receive.shallowupdate true &&
7677
(
7778
cd shallow2 &&
78-
git push ../.git +main:refs/remotes/shallow2/main
79+
git -c push.shallowExcludeBoundary=false \
80+
push ../.git +main:refs/remotes/shallow2/main
7981
) &&
8082
git log --format=%s shallow2/main >actual &&
8183
git fsck &&
@@ -90,7 +92,8 @@ test_expect_success 'push from shallow to shallow' '
9092
(
9193
cd shallow &&
9294
git --git-dir=../shallow2/.git config receive.shallowupdate true &&
93-
git push ../shallow2/.git +main:refs/remotes/shallow/main &&
95+
git -c push.shallowExcludeBoundary=false \
96+
push ../shallow2/.git +main:refs/remotes/shallow/main &&
9497
git --git-dir=../shallow2/.git config receive.shallowupdate false
9598
) &&
9699
(
@@ -164,4 +167,151 @@ test_expect_success 'push new commit from shallow clone has good deltas' '
164167
test_region pack-objects path-walk config-push.txt
165168
'
166169

170+
test_expect_success 'shallow push only pushes what is necessary' '
171+
git init adv-origin &&
172+
# The shallow grafts are intentionally untagged so that no
173+
# advertised ref points at them.
174+
test_commit --no-tag -C adv-origin a &&
175+
test_commit --no-tag -C adv-origin b &&
176+
177+
git clone --depth=1 "file://$(pwd)/adv-origin" adv-client &&
178+
179+
# The remote branch advances past the history we have, so its
180+
# advertised tip is something we cannot use as a negative tip;
181+
# only the shallow graft lets us exclude the full tree.
182+
test_commit --no-tag -C adv-origin c &&
183+
184+
git -C adv-client checkout -b topic &&
185+
test_commit --no-tag -C adv-client new &&
186+
GIT_PROGRESS_DELAY=0 git -C adv-client push --progress origin topic 2>err &&
187+
188+
# Only the new commit, its tree, and the new blob are sent; sending
189+
# the full tree is avoided by excluding the shallow graft.
190+
test_grep "Enumerating objects: 4, done." err
191+
'
192+
193+
test_expect_success 'push.shallowExcludeBoundary=false sends full tree' '
194+
git init adv-origin2 &&
195+
test_commit --no-tag -C adv-origin2 a &&
196+
test_commit --no-tag -C adv-origin2 b &&
197+
198+
git clone --depth=1 "file://$(pwd)/adv-origin2" adv-client2 &&
199+
test_commit --no-tag -C adv-origin2 c &&
200+
201+
git -C adv-client2 checkout -b topic &&
202+
test_commit --no-tag -C adv-client2 new &&
203+
GIT_PROGRESS_DELAY=0 git -C adv-client2 \
204+
-c push.shallowExcludeBoundary=false \
205+
push --progress origin topic 2>err &&
206+
207+
# With the optimization disabled and no advertised ref pointing at
208+
# the shallow graft, the full snapshot down to the shallow graft is
209+
# resent, including its full tree.
210+
test_grep "Enumerating objects: 7, done." err
211+
'
212+
213+
# A rejected ref must not over-exclude objects that another, accepted ref
214+
# legitimately needs in the pack. Set up a testcase using two independent
215+
# shallow roots.
216+
#
217+
# origin: two unrelated histories; only branch A carries blob O (sh=shared)
218+
# A: A0---A1 (A0, A1 trees contain sh=O)
219+
# B: B0---B1 (no "shared" blob)
220+
#
221+
# receiver: seeded from branch B only, under both ref names; lacks blob O
222+
# refs/heads/B -> B1
223+
# refs/heads/A -> B1 (makes our A push a non-fast-forward)
224+
#
225+
# client: "clone --depth=1 --no-single-branch" gives a graft at each tip
226+
# and a copy of blob O under A1 (x = cut parents = shallow graft)
227+
# x x
228+
# | |
229+
# A1 B1
230+
# | |
231+
# cX topic=cY (cY re-adds sh=O, which the receiver lacks)
232+
#
233+
# push "A topic" (non-atomic):
234+
# A -> a non-fast-forward vs receiver A=B1, so its ref update is
235+
# rejected locally and never applied. It still takes part in
236+
# the shared pack computation, and the buggy code also walked
237+
# back from it to graft A1 (which owns O).
238+
# topic -> accepted; cY grafts onto B1 and needs blob O.
239+
#
240+
# Using the shallow graft A1 (an ancestor of A) to trim the pack, even
241+
# though our push of A is rejected locally, would omit blob O from topic's
242+
# pack -- yet topic needs O. We want to ensure that when topic is pushed,
243+
# O is sent along with it despite A being rejected.
244+
test_expect_success 'shallow push does not over-exclude for an accepted ref via a rejected one' '
245+
# origin
246+
git init tworoot-origin &&
247+
git -C tworoot-origin checkout -b A &&
248+
test_commit -C tworoot-origin --no-tag has-shared sh shared &&
249+
test_commit -C tworoot-origin --no-tag A1 &&
250+
git -C tworoot-origin switch --orphan B &&
251+
test_commit -C tworoot-origin --no-tag B0 &&
252+
test_commit -C tworoot-origin --no-tag B1 &&
253+
254+
# receiver: branch B only, exposed as both B and A
255+
git init --bare tworoot-receiver.git &&
256+
git -C tworoot-origin push "file://$(pwd)/tworoot-receiver.git" \
257+
B:refs/heads/B B:refs/heads/A &&
258+
259+
# client: a shallow graft at each branch tip
260+
git clone --depth=1 --no-single-branch \
261+
"file://$(pwd)/tworoot-origin" tworoot-client &&
262+
263+
# branch A gets commit cX; including A in the push gives us a
264+
# locally-rejected ref whose graft A1 the buggy code walked to. The A
265+
# ref update is a non-fast-forward, so it is rejected and never applied.
266+
git -C tworoot-client checkout A &&
267+
test_commit -C tworoot-client --no-tag cX &&
268+
269+
# branch topic is what we actually send, reintroducing blob O on B1
270+
git -C tworoot-client checkout -b topic B &&
271+
test_commit -C tworoot-client --no-tag reintroduce sh shared &&
272+
273+
# push both in one command: they share a single pack computation, so a
274+
# graft reached from the rejected A can strip objects that topic needs.
275+
# The A ref update is rejected locally (non-fast-forward); the shared
276+
# pack must still contain blob O for topic to land on the receiver.
277+
test_must_fail git -C tworoot-client push \
278+
"file://$(pwd)/tworoot-receiver.git" A topic &&
279+
git --git-dir=tworoot-receiver.git rev-parse --verify topic
280+
'
281+
282+
# push.shallowExcludeBoundary (default true) omits the shallow boundary
283+
# snapshot from the pack, since an ordinary receiver already has it. The
284+
# exception is a receiver willing to adopt a *new* shallow root
285+
# (receive.shallowUpdate): it genuinely needs that snapshot, so the default
286+
# optimization leaves it unable to graft the new root. Verify the receiver
287+
# rejects such a push (rather than corrupting itself), and that setting the
288+
# config to false restores the full snapshot and lets the push succeed. This
289+
# is the tradeoff that motivates the config knob.
290+
test_expect_success 'default push to a shallowUpdate receiver rejects a rootless snapshot' '
291+
git init seed-origin &&
292+
test_commit -C seed-origin s1 &&
293+
test_commit -C seed-origin s2 &&
294+
test_commit -C seed-origin s3 &&
295+
296+
# depth-2: a shallow graft at s2, pushing s3 on top of it
297+
git clone --depth=2 "file://$(pwd)/seed-origin" seed-client &&
298+
299+
git init --bare seed-receiver.git &&
300+
git --git-dir=seed-receiver.git config receive.shallowUpdate true &&
301+
302+
# Default (optimization on): the s2 boundary snapshot is withheld, so
303+
# the receiver cannot graft the new root and rejects the push, leaving
304+
# the ref uncreated.
305+
test_must_fail git -C seed-client push \
306+
"file://$(pwd)/seed-receiver.git" HEAD:refs/heads/seeded 2>err &&
307+
test_grep "remote rejected" err &&
308+
test_must_fail git --git-dir=seed-receiver.git rev-parse --verify seeded &&
309+
310+
# Opt-out: the full snapshot is sent, so the same push now succeeds and
311+
# the new shallow root is grafted.
312+
git -C seed-client -c push.shallowExcludeBoundary=false push \
313+
"file://$(pwd)/seed-receiver.git" HEAD:refs/heads/seeded &&
314+
git --git-dir=seed-receiver.git rev-parse --verify seeded
315+
'
316+
167317
test_done

0 commit comments

Comments
 (0)