The References module in Drupal 7 allows for easy creation and removal of user and node references through Drupal's interface. However, programmatically adding and removing these references is a little more difficult.
You basically have to load the node which has the reference in it, edit the reference field (in my example, the reference field can have an unlimited number of references), add or remove the user ID (or node ID if you're chaging a node reference), and save the node.
Let's look at the example of simply adding a user reference to a node:
$node = node_load($nid);
// Add the user ID you'd like to add to this node reference.
$node->field_node_user_references[$node->language] = array('uid' => $uid);
// Save the node.
It takes a little more effort to remove a user reference (or node reference) from a node. For this, since I have to do it for a few different fields on a node, I've written a helper function that removes a given $uid from the array of user references on a given node.