Skip to content

[WIP] First test auto adaptive #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jiachend wants to merge 6 commits into
base: main
Choose a base branch
from
Open

[WIP] First test auto adaptive #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
16d3307
Keep threshold always positive
jiachend Apr 26, 2022
c7514c0
Keep threshold always positive
jiachend Apr 26, 2022
4886665
Merge pull request #13 from lguerdan/keep-positive-threshold
lguerdan Apr 26, 2022
659edff
Add params recommendation to exp3_params
jiachend Apr 26, 2022
b7cb46e
Merge pull request #15 from lguerdan/param-rec
lguerdan Apr 28, 2022
14c2212
First test auto adaptive
jiachend Apr 28, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions experiments.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,10 +82,10 @@ def run_central_baseline_adaptive_cuttoff(params, clipping_methods):
'clipping': 'Linear',
'num_microbatches': 32,
'batch_size': 32,
'S': 1,
'z': 0.2,
'S': 5,
'z': 0.1,
'gamma': 0.7,
'lr_c': 0.1,
'lr_c': 0.01,
'momentum': 0.5,
'decay': 0,
'n_epochs': 6,
Expand Down
15 changes: 10 additions & 5 deletions train.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,7 @@ def train(
momentum = opt_params['momentum']
decay = opt_params['decay']
S_e = opt_params['S']
gamma = opt_params['gamma']

criterion = nn.CrossEntropyLoss(reduction='none')
optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=momentum, weight_decay=decay)
Expand All @@ -116,12 +117,13 @@ def train(
'train_acc': [],
'test_loss': [],
'test_acc': [],
'S': []
'S': [],
'gamma': [],
}

print(f"Training {n_epochs} epoch(s) w/ {len(trainloader)} batches each.", flush=True)
for epoch in range(n_epochs):
train_loss, train_acc, S_e = train_epoch(model, trainloader, device, optimizer, criterion, S_e, opt_params)
train_loss, train_acc, S_e, gamma = train_epoch(model, trainloader, device, optimizer, criterion, S_e, gamma, opt_params)
test_loss, test_acc = test(model, testloader, device)

# Write training metrics
Expand All @@ -138,6 +140,7 @@ def train(
log['test_loss'].append(test_loss)
log['test_acc'].append(test_acc)
log['S'].append(S_e)
log['gamma'].append(gamma)

# export scalar data to JSON for external processing
helpers.write_logs(exp_name, log, opt_params)
Expand All @@ -151,16 +154,16 @@ def train_epoch(
optimizer: torch.optim,
criterion,
S_e,
gamma,
opt_params,
) -> List[Tuple[float, float]]:
# DP-SGD parameters
adaptive_clipping = opt_params['clipping']
num_microbatches = opt_params['num_microbatches']
S = S_e
z = opt_params['z']
gamma = opt_params['gamma'] # Target quantile
lr_c = opt_params['lr_c']
sigma_b = 1.1 # Test value for sigma used in adaptive clipping
sigma_b = 1.5 # Test value for sigma used in adaptive clipping
sigma = z * S

# Define loss and optimizer
Expand Down Expand Up @@ -198,6 +201,7 @@ def train_epoch(
saved_var[tensor_name].add_(new_grad)
model.zero_grad()

gamma += (b/num_microbatches-gamma)/2

@lguerdan lguerdan Apr 29, 2022

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’ve been thinking about this more. Intuitively, we want to favor a clipping threshold at the beginning of training that has low bias, but higher variance. In this case, it’s ok that we add a higher clipping threshold in exchange for more noise because the low bias in the estimate helps the model converge towards a good initial solution. However, later on, its important that there is little noise added to the parameter estimate (low variance) so that the model can learn finer grained details.

See discussion by Andrew et. al, for example
image

@lguerdan lguerdan Apr 29, 2022

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One idea is to taper the gamma value over training (akin to a learning rate scheduler). It would be cool if we could somehow estimate the bias / variance trade off during training. One way is to look at the delta between training and validation loss, which is normally taken as a measure of overfitting. I'll try the gamma tapering approach tomorrow morning

if adaptive_clipping == 'Linear':
b += torch.randn(1) * sigma_b
b_t = b / num_microbatches
Expand All @@ -222,8 +226,9 @@ def train_epoch(
total += y.size(0)
correct += (predicted == y).sum().item()
S_e = S_e.item() if adaptive_clipping != 'Fixed' else S_e
S_e = max(S_e, 0.0)

return running_loss / total, correct / total, S_e
return running_loss / total, correct / total, S_e, gamma


def test(
Expand Down