This configures System Preferences > Bluetooth > Advanced…
To uncheck “Open Bluetooth Setup Assistant at startup if no keyboard is detected”, run:
sudo defaults write /Library/Preferences/com.apple.Bluetooth.plist BluetoothAutoSeekKeyboard false
To uncheck “Open Bluetooth Setup Assistant at startup if no mouse or trackpad is detected”, run:
sudo defaults write /Library/Preferences/com.apple.Bluetooth.plist BluetoothAutoSeekPointingDevice
Still trying to figure out a command to disable the third option, “Allow Bluetooth devices to wake this computer”.