Question: I've always wondered why processors stopped at 32 registers. It's by far the fastest piece of the machine, why not just make bigger processors with more registers? Wouldn't that mean less going to the RAM?

First, not all processor architectures stopped at 32 registers. Almost all the RISC architectures that have 32 registers exposed in the instruction set actually have 32 integer registers and 32 more floating point registers (so 64).  (Floating point "add" uses different registers than integer "add".)  The SPARC architecture has register windows.  On the SPARC you can only access 32 integer registers at a time, but the registers act like a stack and you can push and pop new registers 16 at a time.  The Itanium architecture from HP/Intel had 128 integer and 128 floating point registers exposed in the instruction set.  Modern GPUs from NVidia, AMD, Intel, ARM and Imagination Technologies, all expose massive numbers of registers in their register files.  (I know this to be true of the NVidia and Intel architectures, I am not very familiar with the AMD, ARM and Imagination instruction sets, but I think the register files are large there too.)
Second, most modern microprocessors implement register renaming to eliminate unnecessary serialization caused by needing to reuse resources, so the underlying physical register files can be larger (96, 128 or 192 registers on some machines.)  This (and dynamic scheduling) eliminates some of the need for the compiler to generate so many unique register names, while still providing a larger register file to the scheduler.
There are two reasons why it might be difficult to further increase the number of registers exposed in the instruction set.  First, you need to be able to specify the register identifiers in each instruction.  32 registers require a 5 bit register specifier, so 3-address instructions (common on RISC architectures) spend 15 of the 32 instruction bits just to specify the registers.  If you increased that to 6 or 7 bits, then you would have less space to specify opcodes and constants.  GPUs and Itanium have much larger instructions.  Larger instructions come at a cost: you need to use more instruction memory, so your instruction cache behavior is less ideal.
The second reason is access time.  The larger you make a memory the slower it is to access data from it.  (Just in terms of basic physics: the data is stored in 2-dimensional space, so if you are storing $n$ bits, the average distance to a specific bit is $O(\sqrt{n})$.)  A register file is just a small multi-ported memory, and one of the constraints on making it larger is that eventually you would need to start clocking your machine slower to accommodate the larger register file.  Usually in terms of total performance this is a lose.


Question:
I'm training a neural network but the training loss doesn't decrease. How can I fix this?
I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my network's performance doesn't improve on the training set.
A specific variant of this problem arises when the loss has a steep initial decrease and then stops improving almost immediately. Often, this happens because the model is fitting some constant to the target (dependent variable, outcome).

For a regression task that is minimizing the square error (MSE loss), this constant is usually something close to $\bar y$ the mean of the target  (dependent variable, outcome).
For a classification task, it's slightly more subtle, but it can happen that the model fits a constant to predict the proportion of each outcome. Consider the binary classification task with cross-entropy loss. An optimal constant to choose is a solution to $c \log c + (1 - c) \log(1-c)=-\bar y$ where $0 < c< 1$.

In both cases, we would prefer the network to be more specific in the sense that the prediction varies as some function of the features (input, independent variable); we don't need to use a neural network to compute the mean of the response.

This question is intentionally general so that other questions about how to train a neural network can be closed as a duplicate of this one, with the attitude that "if you give a man a fish you feed him for a day, but if you teach a man to fish, you can feed him for the rest of his life." See this Meta thread for a discussion: What's the best way to answer "my neural network doesn't work, please fix" questions?
If your neural network does not generalize well, see: What should I do when my neural network doesn't generalize well?

1.  Verify that your code is bug free
There's a saying among writers that "All writing is re-writing" -- that is, the greater part of writing is revising. For programmers (or at least data scientists) the expression could be re-phrased as "All coding is debugging."
Any time you're writing code, you need to verify that it works as intended. The best method I've ever found for verifying correctness is to break your code into small segments, and verify that each segment works. This can be done by comparing the segment output to what you know to be the correct answer. This is called unit testing. Writing good unit tests is a key piece of becoming a good statistician/data scientist/machine learning expert/neural network practitioner. There is simply no substitute.
You have to check that your code is free of bugs before you can tune network performance! Otherwise, you might as well be re-arranging deck chairs on the RMS Titanic.
There are two features of neural networks that make verification even more important than for other types of machine learning or statistical models.

Neural networks are not "off-the-shelf" algorithms in the way that random forest or logistic regression are. Even for simple, feed-forward networks, the onus is largely on the user to make numerous decisions about how the network is configured, connected, initialized and optimized. This means writing code, and writing code means debugging.

Even when a neural network code executes without raising an exception, the network can still have bugs! These bugs might even be the insidious kind for which the network will train, but get stuck at a sub-optimal solution, or the resulting network does not have the desired architecture. (This is an example of the difference between a syntactic and semantic error.)

This Medium post, "How to unit test machine learning code," by Chase Roberts discusses unit-testing for machine learning models in more detail. I borrowed this example of buggy code from the article:
def make_convnet(input_image):
    net = slim.conv2d(input_image, 32, [11, 11], scope="conv1_11x11")
    net = slim.conv2d(input_image, 64, [5, 5], scope="conv2_5x5")
    net = slim.max_pool2d(net, [4, 4], stride=4, scope='pool1')
    net = slim.conv2d(input_image, 64, [5, 5], scope="conv3_5x5")
    net = slim.conv2d(input_image, 128, [3, 3], scope="conv4_3x3")
    net = slim.max_pool2d(net, [2, 2], scope='pool2')
    net = slim.conv2d(input_image, 128, [3, 3], scope="conv5_3x3")
    net = slim.max_pool2d(net, [2, 2], scope='pool3')
    net = slim.conv2d(input_image, 32, [1, 1], scope="conv6_1x1")
    return net

Do you see the error? Many of the different operations are not actually used because previous results are over-written with new variables. Using this block of code in a network will still train and the weights will update and the loss might even decrease -- but the code definitely isn't doing what was intended.  (The author is also inconsistent about using single- or double-quotes but that's purely stylistic.)
The most common programming errors pertaining to neural networks are

Variables are created but never used (usually because of copy-paste errors);
Expressions for gradient updates are incorrect;
Weight updates are not applied;
Loss functions are not measured on the correct scale (for example, cross-entropy loss can be expressed in terms of probability or logits)
The loss is not appropriate for the task (for example, using categorical cross-entropy loss for a regression task).
Dropout is used during testing, instead of only being used for training.
Make sure you're minimizing the loss function $L(x)$, instead of minimizing $-L(x)$.
Make sure your loss is computed correctly.

Unit testing is not just limited to the neural network itself. You need to test all of the steps that produce or transform data and feed into the network. Some common mistakes here are

NA or NaN or Inf values in your data creating NA or NaN or Inf values in the output, and therefore in the loss function.
Shuffling the labels independently from the samples (for instance, creating train/test splits for  the labels and samples separately);
Accidentally assigning the training data as the testing data;
When using a train/test split, the model references the original, non-split data instead of the training partition or the testing partition.
Forgetting to scale the testing data;
Scaling the testing data using the statistics of the test partition instead of the train partition;
Forgetting to un-scale the predictions (e.g. pixel values are in [0,1] instead of [0, 255]).
Here's an example of a question where the problem appears to be one of model configuration or hyperparameter choice, but actually the problem was a subtle bug in how gradients were computed. Is this drop in training accuracy due to a statistical or programming error?

2. For the love of all that is good, scale your data
The scale of the data can make an enormous difference on training. Sometimes, networks simply won't reduce the loss if the data isn't scaled. Other networks will decrease the loss, but only very slowly. Scaling the inputs (and certain times, the targets) can dramatically improve the network's training.

Prior to presenting data to a neural network, standardizing the data to have 0 mean and unit variance, or to lie in a small interval like $[-0.5, 0.5]$ can improve training. This amounts to pre-conditioning, and removes the effect that a choice in units has on network weights. For example, length in millimeters and length in kilometers both represent the same concept, but are on different scales. The exact details of how to standardize the data depend on what your data look like.

Data normalization and standardization in neural networks

Why does $[0,1]$ scaling dramatically increase training time for feed forward ANN (1 hidden layer)?

Batch or Layer normalization can improve network training. Both seek to improve the network by keeping a running mean and standard deviation for neurons' activations as the network trains. It is not well-understood why this helps training, and remains an active area of research.

"Understanding Batch Normalization" by Johan Bjorck, Carla Gomes, Bart Selman
"Towards a Theoretical Understanding of Batch Normalization" by Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Ming Zhou, Klaus Neymeyr, Thomas Hofmann
"How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift)" by Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry

3. Crawl Before You Walk; Walk Before You Run
Wide and deep neural networks, and neural networks with exotic wiring, are the Hot Thing right now in machine learning. But these networks didn't spring fully-formed into existence; their designers built up to them from smaller units. First, build a small network with a single hidden layer and verify that it works correctly. Then incrementally add additional model complexity, and verify that each of those works as well.

Too few neurons in a layer can restrict the representation that the network learns, causing under-fitting. Too many neurons can cause over-fitting because the network will "memorize" the training data.
Even if you can prove that there is, mathematically, only a small number of neurons necessary to model a problem, it is often the case that having "a few more" neurons makes it easier for the optimizer to find a "good" configuration. (But I don't think anyone fully understands why this is the case.) I provide an example of this in the context of the XOR problem here: Aren't my iterations needed to train NN for XOR with MSE < 0.001 too high?.

Choosing the number of hidden layers lets the network learn an abstraction from the raw data. Deep learning is all the rage these days, and networks with a large number of layers have shown impressive results. But adding too many hidden layers can make risk overfitting or make it very hard to optimize the network.

Choosing a clever network wiring can do a lot of the work for you. Is your data source amenable to specialized network architectures? Convolutional neural networks can achieve impressive results on "structured" data sources, image or audio data. Recurrent neural networks can do well on sequential data types, such as natural language or time series data. Residual connections can improve deep feed-forward networks.

4. Neural Network Training Is Like Lock Picking
To achieve state of the art, or even merely good, results, you have to set up all of the parts configured to work well together. Setting up a neural network configuration that actually learns is a lot like picking a lock: all of the pieces have to be lined up just right. Just as it is not sufficient to have a single tumbler in the right place, neither is it sufficient to have only the architecture, or only the optimizer, set up correctly.
Tuning configuration choices is not really as simple as saying that one kind of configuration choice (e.g. learning rate) is more or less important than another (e.g. number of units), since all of these choices interact with all of the other choices, so one choice can do well in combination with another choice made elsewhere.
This is a non-exhaustive list of the configuration options which are not also regularization options or numerical optimization options.
All of these topics are active areas of research.

The network initialization is often overlooked as a source of neural network bugs. Initialization over too-large an interval can set initial weights too large, meaning that single neurons have an outsize influence over the network behavior.

The key difference between a neural network and a regression model is that a neural network is a composition of many nonlinear functions, called activation functions. (See: What is the essential difference between neural network and linear regression)
Classical neural network results focused on sigmoidal activation functions (logistic or $\tanh$ functions). A recent result has found that ReLU (or similar) units tend to work better because the have steeper gradients, so updates can be applied quickly. (See: Why do we use ReLU in neural networks and how do we use it?) One caution about ReLUs is the "dead neuron" phenomenon, which can stymie learning; leaky relus and similar variants avoid this problem. See

Why can't a single ReLU learn a ReLU?

My ReLU network fails to launch

There are a number of other options. See: Comprehensive list of activation functions in neural networks with pros/cons

Residual connections are a neat development that can make it easier to train neural networks. "Deep Residual Learning for Image Recognition"
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun In: CVPR. (2016). Additionally, changing the order of operations within the residual block can further improve the resulting network. "Identity Mappings in Deep Residual Networks" by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.

5. Non-convex optimization is hard
The objective function of a neural network is only convex when there are no hidden units, all activations are linear, and the design matrix is full-rank -- because this configuration is identically an ordinary regression problem.
In all other cases, the optimization problem is non-convex, and non-convex optimization is hard. The challenges of training neural networks are well-known (see: Why is it hard to train deep neural networks?). Additionally, neural networks have a very large number of parameters, which restricts us to solely first-order methods (see: Why is Newton's method not widely used in machine learning?). This is a very active area of research.

Setting the learning rate too large will cause the optimization to diverge, because you will leap from one side of the "canyon" to the other. Setting this too small will prevent you from making any real progress, and possibly allow the noise inherent in SGD to overwhelm your gradient estimates. See:

How can change in cost function be positive?

Gradient clipping re-scales the norm of the gradient if it's above some threshold. I used to think that this was a set-and-forget parameter, typically at 1.0, but I found that I could make an LSTM language model dramatically better by setting it to 0.25. I don't know why that is.

Learning rate scheduling can decrease the learning rate over the course of training. In my experience, trying to use scheduling is a lot like regex: it replaces one problem ("How do I get learning to continue after a certain epoch?") with two problems ("How do I get learning to continue after a certain epoch?" and "How do I choose a good schedule?"). Other people insist that scheduling is essential. I'll let you decide.

Choosing a good minibatch size can influence the learning process indirectly, since a larger mini-batch will tend to have a smaller variance (law-of-large-numbers) than a smaller mini-batch. You want the mini-batch to be large enough to be informative about the direction of the gradient, but small enough that SGD can regularize your network.

There are a number of variants on stochastic gradient descent which use momentum, adaptive learning rates, Nesterov updates and so on to improve upon vanilla SGD. Designing a better optimizer is very much an active area of research. Some examples:

No change in accuracy using Adam Optimizer when SGD works fine
How does the Adam method of stochastic gradient descent work?
Why does momentum escape from a saddle point in this famous image?

When it first came out, the Adam optimizer generated a lot of interest. But some recent research has found that SGD with momentum can out-perform adaptive gradient methods for neural networks. "The Marginal Value of Adaptive Gradient Methods in Machine Learning" by Ashia C. Wilson, Rebecca Roelofs, Mitchell Stern, Nathan Srebro, Benjamin Recht

But on the other hand, this very recent paper proposes a new adaptive learning-rate optimizer which supposedly closes the gap between adaptive-rate methods and SGD with momentum. "Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks" by Jinghui Chen, Quanquan Gu

Adaptive gradient methods, which adopt historical gradient information to automatically adjust the learning rate, have been observed to generalize worse than stochastic gradient descent (SGD) with momentum in training deep neural networks. This leaves how to close the generalization gap of adaptive gradient methods an open problem. In this work, we show that adaptive gradient methods such as Adam, Amsgrad, are sometimes "over adapted". We design a new algorithm, called Partially adaptive momentum estimation method (Padam), which unifies the Adam/Amsgrad with SGD to achieve the best from both worlds. Experiments on standard benchmarks show that Padam can maintain fast convergence rate as Adam/Amsgrad while generalizing as well as SGD in training deep neural networks. These results would suggest practitioners pick up adaptive gradient methods once again for faster training of deep neural networks.

Specifically for triplet-loss models, there are a number of tricks which can improve training time and generalization. See: In training a triplet network, I first have a solid drop in loss, but eventually the loss slowly but consistently increases. What could cause this?

6. Regularization
Choosing and tuning network regularization is a key part of building a model that generalizes well (that is, a model that is not overfit to the training data). However, at the time that your network is struggling to decrease the loss on the training data -- when the network is not learning -- regularization can obscure what the problem is.
When my network doesn't learn, I turn off all regularization and verify that the non-regularized network works correctly. Then I add each regularization piece back, and verify that each of those works along the way.
This tactic can pinpoint where some regularization might be poorly set. Some examples are

$L^2$ regularization (aka weight decay) or $L^1$ regularization is set too large, so the weights can't move.

Two parts of regularization are in conflict. For example, it's widely observed that layer normalization and dropout are difficult to use together. Since either on its own is very useful, understanding how to use both is an active area of research.

"Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift" by Xiang Li, Shuo Chen, Xiaolin Hu, Jian Yang
"Adjusting for Dropout Variance in Batch Normalization and Weight Initialization" by Dan Hendrycks, Kevin Gimpel.
"Self-Normalizing Neural Networks" by Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter

7. Keep a Logbook of Experiments
When I set up a neural network, I don't hard-code any parameter settings. Instead, I do that in a configuration file (e.g., JSON) that is read and used to populate network configuration details at runtime. I keep all of these configuration files. If I make any parameter modification, I make a new configuration file. Finally, I append as comments all of the per-epoch losses for training and validation.
The reason that I'm so obsessive about retaining old results is that this makes it very easy to go back and review previous experiments. It also hedges against mistakenly repeating the same dead-end experiment. Psychologically, it also lets you look back and observe "Well, the project might not be where I want it to be today, but I am making progress compared to where I was $k$ weeks ago."
As an example, I wanted to learn about LSTM language models, so I decided to make a Twitter bot that writes new tweets in response to other Twitter users. I worked on this in my free time, between grad school and my job. It took about a year, and I iterated over about 150 different models before getting to a model that did what I wanted: generate new English-language text that (sort of) makes sense. (One key sticking point, and part of the reason that it took so many attempts, is that it was not sufficient to simply get a low out-of-sample loss, since early low-loss models had managed to memorize the training data, so it was just reproducing germane blocks of text verbatim in reply to prompts -- it took some tweaking to make the model more spontaneous and still have low loss.)


Question:
I am trying to design a cloth that, from the point of view of a camera, is very difficult to compress with JPG, resulting in big-size files (or leading to low image quality if file size is fixed).
It must work even if the cloth is far away from the camera, or rotated (let's say the scale can vary from 1x to 10x).
Noise is quite good (hard to compress), but it becomes grey when looking from far, becoming easy to compress. A good pattern would be kind of fractal, looking similar at all scales.
Foliage is better (leaves, tiny branches, small branches, big branches), but it uses too few colors.
Here is a first try:

I am sure there are more optimum patterns.
Maybe hexagon or triangle tessellations would perform better.
JPG uses the Y′ Cb Cr color space, I think Cb Cr can be generated in a similar way, but I guess it's better to not use uniformly the full scope of Y' (brightness) since camera will saturate the bright or dark areas (lighting is never perfect).
QUESTION: What is the optimum cloth pattern for this problem?

Noise is quite good (hard to compress), but it becomes grey when looking from far, becoming easy to compress. A good pattern would be kind of fractal, looking similar at all scales.

Well, there is fractal noise.  I think Brownian noise is fractal, looking the same as you zoom into it.  Wikipedia talks about adding Perlin noise to itself at different scales to produce fractal noise, which is maybe identical, I'm not sure:

I don't think this would be hard to compress, though.  Noise is hard for lossless compression, but JPEG is lossy, so it's just going to throw away the detail instead of struggling with it.  I'm not sure if it's possible to make something "hard for JPEG to compress" since it will just ignore anything that's too hard to compress at that quality level.
Something with hard edges at any scale would probably be better, like the infinite checkerboard plane:

Also something with lots of colors.  Maybe look at actual fractals instead of fractal noise.  Maybe a Mondrian fractal?  :)


Question:
Red-green colorblindness seems to make it harder for a hunter-gatherer to see whether a fruit is ripe and thus worth picking.
Is there a reason why selection hasn't completely removed red-green color blindness? Are there circumstances where this trait provides an evolutionary benefit?

Short answer
Color-blind subjects are better at detecting color-camouflaged objects. This may give color blinds an advantage in terms of spotting hidden dangers (predators) or finding camouflaged foods.
Background
There are two types of red-green blindness: protanopia (red-blind) and deuteranopia (green-blind), i.e., these people miss one type of cone, namely the (red L cone or the green M cone).
These conditions should be set apart from the condition where there are mutations in the L cones shifting their sensitivity to the green cone spectrum (deuteranomaly) or vice versa (protanomaly).
Since you are talking color-"blindness", as opposed to reduced sensitivity to red or green, I reckon you are asking about true dichromats, i.e., protanopes and deuteranopes. It's an excellent question as to why 2% of the men have either one condition, given that:
Protanopes are more likely to confuse:-

Black with many shades of red
Dark brown with dark green, dark orange and dark red
Some blues with some reds, purples and dark pinks
Mid-greens with some oranges

Deuteranopes are more likely to confuse:-

Mid-reds with mid-greens
Blue-greens with grey and mid-pinks
Bright greens with yellows
Pale pinks with light grey
Mid-reds with mid-brown
Light blues with lilac

There are reports on the benefits of being red-green color blind under certain specific conditions. For example, Morgan et al. (1992) report that the identification of a target area with a different texture or orientation pattern was performed better by dichromats when the surfaces were painted with irrelevant colors. In other words, when color is simply a distractor and confuses the subject to focus on the task (i.e., texture or orientation discrimination), the lack of red-green color vision can actually be beneficial. This in turn could be interpreted as dichromatic vision being beneficial over trichromatic vision to detect color-camouflaged objects.
Reports on improved foraging of dichromats under low-lighting are debated, but cannot be excluded. The better camouflage-breaking performance of dichromats is, however, an established phenomenon (Cain et al., 2010).
During the Second World War it was suggested that color-deficient observers could often penetrate camouflage that deceived the normal observer. The idea has been a recurrent one, both with respect to military camouflage and with respect to the camouflage of the natural world (reviewed in Morgan et al. (1992)
Outlines, rather than colors, are responsible for pattern recognition. In the military, colorblind snipers and spotters are highly valued for these reasons (source: De Paul University). If you sit back far from your screen, look at the normal full-color picture on the left and compare it to the dichromatic picture on the right; the picture on the right appears at higher contrast in trichromats, but dichromats may not see any difference between the two:

Left: full-color image, right: dichromatic image. source: De Paul University
However, I think the dichromat trait is simply not selected against strongly and this would explain its existence more easily than finding reasons it would be selected for (Morgan et al., 1992).
References
- Cain et al., Biol Lett (2010); 6, 3–38
- Morgan et al., Proc R Soc B (1992); 248: 291-5


Question:
TL:DR: Is it ever a good idea to train an ML model on all the data available before shipping it to production? Put another way, is it ever ok to train on all data available and not check if the model overfits, or get a final read of the expected performance of the model?

Say I have a family of models parametrized by $\alpha$. I can do a search (e.g. a grid search) on $\alpha$ by, for example, running k-fold cross-validation for each candidate.
The point of using cross-validation for choosing $\alpha$ is that I can check if a learned model $\beta_i$ for that particular $\alpha_i$ had e.g. overfit, by testing it on the "unseen data" in each CV iteration (a validation set). After iterating through all $\alpha_i$'s, I could then choose a model $\beta_{\alpha^*}$ learned for the parameters $\alpha^*$ that seemed to do best on the grid search, e.g. on average across all folds.
Now, say that after model selection I would like to use all the the data that I have available in an attempt to ship the best possible model in production.  For this, I could use the parameters $\alpha^*$ that I chose via grid search with cross-validation, and then, after training the model on the full ($F$) dataset, I would a get a single new learned model $\beta^{F}_{\alpha^*}$
The problem is that, if I use my entire dataset for training,  I can't reliably check if this new learned model $\beta^{F}_{\alpha^*}$ overfits or how it may perform on unseen data. So is this at all good practice? What is a good way to think about this problem?

The way to think of cross-validation is as estimating the performance obtained using a method for building a model, rather than for estimating the performance of a model.
If you use cross-validation to estimate the hyperparameters of a model (the $\alpha$s) and then use those hyper-parameters to fit a model to the whole dataset, then that is fine, provided that you recognise that the cross-validation estimate of performance is likely to be (possibly substantially) optimistically biased.  This is because part of the model (the hyper-parameters) have been selected to minimise the cross-validation performance, so if the cross-validation statistic has a non-zero variance (and it will) there is the possibility of over-fitting the model selection criterion.
If you want to choose the hyper-parameters and estimate the performance of the resulting model then you need to perform a nested cross-validation, where the outer cross-validation is used to assess the performance of the model, and in each fold cross-validation is used to determine the hyper-parameters separately in each fold.  You build the final model by using cross-validation on the whole set to choose the hyper-parameters and then build the classifier on the whole dataset using the optimized hyper-parameters.
This is of course computationally expensive, but worth it as the bias introduced by improper performance estimation can be large.  See my paper
G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. ( pdf)
However, it is still possible to have over-fitting in model selection (nested cross-validation just allows you to test for it).  A method I have found useful is to add a regularisation term to the cross-validation error that penalises hyper-parameter values likely to result in overly-complex models, see
G. C. Cawley and N. L. C. Talbot, Preventing over-fitting in model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007. (
So the answers to your question are (i) yes, you should use the full dataset to produce your final model as the more data you use the more likely it is to generalise well but (ii) make sure you obtain an unbiased performance estimate via nested cross-validation and potentially consider penalising the cross-validation statistic to further avoid over-fitting in model selection.


Question:
A diode is put in parallel with a relay coil (with opposite polarity) to prevent damage to other components when the relay is turned off.
Here's an example schematic I found online:

I'm planning on using a relay with a coil voltage of 5V and contact rating of 10A.
How do I determine the required specifications for the diode, such as voltage, current, and switching time?

First determine the coil current when the coil is on. This is the current that will flow through the diode when the coil is switched off. In your relay, the coil current is shown as 79.4 mA. Specify a diode for at least 79.4 mA current.  In your case, a 1N4001 current rating far exceeds the requirement.
The diode reverse voltage rating should be at least the voltage applied to the relay coil. Normally a designer puts in plenty of reserve in the reverse rating. A diode in your application having 50 volts would be more than adequate. Again 1N4001 will do the job.
Additionally, the 1N4007 (in single purchase quantities) costs the same but has 1000 volt rating.


Question: Sometimes men wake up with an erection in the morning. Why does this happen?

Sometimes men wake up with an erection in the morning. Why does this happen?
Shortly speaking: REM (Rapid Eye Movement) is one phase of sleep. During this phase, we dream and some of our neurotransmitters are shut off. This include norepinephrine, which is involved in controlling erections. Norepinephrine prevents blood from entering the penis (preventing the erection). In absence of norepinephrine—during REM phase norepinephrine is absent—blood enters the penis, leading to an erection. This phenomenon is called nocturnal penile tumescence. Such erections typically occur 3 to 5 times a night. A related question concerning similar erections in women can be found here.
High pressure in the bladder may also lead to a "reflex erection". This erection allows for preventing uncontrolled urination. The drawback is that when in the morning one has an erection and can't wait to pee it might get hard to accurately target the toilets!
This video is also a nice and easy source of information on the subject.
Is it bad?
(Reading my "note" below, you have edited your post to get rid of this question, thank you)
It is perfectly healthy you don't have to worry about that. These erections are even thought of as contributing to penile health. At the opposite end of the spectrum, the absence of erections during the nights are an index of Erectile Dysfunction (E.D.).
Note
Be aware that medical questions are often considered off-topic on this site. Asking "is it bad?" turns your question into a medical one. Health-related questions (but not personal health) should be asked on health.SE


Question:
Does anyone have recommendations on a usable, fast C++ matrix library?
What I mean by usable is the following:

Matrix objects have an intuitive interface (ex.: I can use rows and columns while indexing)
I can do anything with the matrix class that I can do with LAPACK and BLAS
Easy to learn and use API
Relatively painless to install in Linux (I use Ubuntu 11.04 right now)

To me, usability is more important than speed or memory usage right now, to avoid premature optimization. In writing the code, I could always use 1-D arrays (or STL vectors) and proper index or pointer arithmetic to emulate a matrix, but I'd prefer not to in order to avoid bugs. I'd also like to focus my mental effort on the actual problem I'm trying to solve and program into the problem domain, rather than use part of my finite attention to remember all of the little programming tricks I used to emulate matrices as arrays, and remember LAPACK commands, et cetera. Plus, the less code I have to write, and the more standardized it is, the better.
Dense versus sparse doesn't matter yet; some of the matrices I am dealing with will be sparse, but not all of them. However, if a particular package handles dense or sparse matrices well, it is worth mentioning.
Templating doesn't matter much to me either, since I'll be working with standard numeric types and don't need to store anything other than doubles, floats, or ints. It's nice, but not necessary for what I'd like to do.

I've gathered the following from online research so far:
I've used Armadillo a little bit, and found the interface to be intuitive enough, and it was easy to locate binary packages for Ubuntu (and I'm assuming other Linux distros). I haven't compiled it from source, but my hope is that it wouldn't be too difficult. It meets most of my design criteria, and uses dense linear algebra. It can call LAPACK or MKL routines. There generally is no need to compile Armadillo, it is a purely template-based library: You just include the header and link to BLAS/LAPACK or MKL etc.
I've heard good things about Eigen, but haven't used it. It claims to be fast, uses templating, and supports dense linear algebra. It doesn't have LAPACK or BLAS as a dependency, but appears to be able to do everything that LAPACK can do (plus some things LAPACK can't). A lot of projects use Eigen, which is promising. It has a binary package for Ubuntu, but as a header-only library it's trivial to use elsewhere too.
The Matrix Template Library version 4 also looks promising, and uses templating. It supports both dense and sparse linear algebra, and can call UMFPACK as a sparse solver. The features are somewhat unclear from their website. It has a binary package for Ubuntu, downloadable from their web site.
PETSc, written by a team at Argonne National Laboratory, has access to sparse and dense linear solvers, so I'm presuming that it can function as a matrix library. It's written in C, but has C++ bindings, I think (and even if it didn't, calling C from C++ is no problem). The documentation is incredibly thorough. The package is a bit overkill for what I want to do now (matrix multiplication and indexing to set up mixed-integer linear programs), but could be useful as a matrix format for me in the future, or for other people who have different needs than I do.
Trilinos, written by a team at Sandia National Laboratory, provides object-oriented C++ interfaces for dense and sparse matrices through its Epetra component, and templated interfaces for dense and sparse matrices through its Tpetra component. It also has components that provide linear solver and eigensolver functionality. The documentation does not seem to be as polished or prominent as PETSc; Trilinos seems like the Sandia analog of PETSc. PETSc can call some of the Trilinos solvers. Binaries for Trilinos are available for Linux.
Blitz is a C++ object-oriented library that has Linux binaries. It doesn't seem to be actively maintained (2012-06-29: a new version has just appeared yesterday!), although the mailing list is active, so there is some community that uses it. It doesn't appear to do much in the way of numerical linear algebra beyond BLAS, and looks like a dense matrix library. It uses templates.
Boost::uBLAS is a C++ object-oriented library and part of the Boost project. It supports templating and dense numerical linear algebra. I've heard it's not particularly fast.
The Template Numerical Toolkit is a C++ object-oriented library developed by NIST. Its author, Roldan Pozo, seems to contribute patches occasionally, but it doesn't seem to be under active development any longer (last update was 2010). It focuses on dense linear algebra, and provides interfaces for some basic matrix decompositions and an eigenvalue solver.
Elemental, developed by Jack Poulson, is a distributed memory (parallel) dense linear algebra software package written in a style similar to FLAME. For a list of features and background on the project, see his documentation. FLAME itself has an associated library for sequential and shared-memory dense linear algebra, called libflame, which appears to be written in object-oriented C. Libflame looks a lot like LAPACK, but with better notation underlying the algorithms to make development of fast numerical linear algebra libraries more of a science and less of a black art.
There are other libraries that can be added to the list; if we're counting sparse linear algebra packages as "matrix libraries", the best free one I know of in C is SuiteSparse, which is programmed in object-oriented style. I've used SuiteSparse and found it fairly easy to pick up; it depends on BLAS and LAPACK for some of the algorithms that decompose sparse problems into lots of small, dense linear algebra subproblems. The lead author of the package, Tim Davis, is incredibly helpful and a great all-around guy.
The Harwell Subroutine Libraries are famous for their sparse linear algebra routines, and are free for academic users, though you have to go through this process of filling out a form and receiving an e-mail for each file that you want to download. Since the subroutines often have dependencies, using one solver might require downloading five or six files, and the process can get somewhat tedious, especially since the form approval is not instantaneous.
There are also other sparse linear algebra solvers, but as far as I can tell, MUMPS and other packages are focused mostly on the solution of linear systems, and solving linear systems is the least of my concerns right now. (Maybe later, I will need that functionality, and it could be useful for others.)


Question:
What the difference between TPM and CPM when dealing with RNA seq data?
What metrics would you use if you have to perform some down stream analysis other than Differential expression for eg.
Clustering analysis using Hclust function and then plotting heat map to find differences in terms of expression levels, correlation and pca
Is it wrong to use TPM for such analysis, if yes then when does one use TPM versus CPM.

You can find the various equations in this oft-cited blog post from Harold Pimentel. CPM is basically depth-normalized counts, whereas TPM is length-normalized (and then normalized by the length-normalized values of the other genes).
If one has to choose between those two choices one typically chooses TPM for most things, since generally the length normalization is handy. Realistically, you probably want log(TPM) since otherwise noise in your most highly expressed genes dominates over small expression signals.


Question:
My textbook mentions that SCUBA tanks often contain a mixture of oxygen and nitrogen along with a little helium which serves as a diluent.
Now as I remember it, divers take care not to surface too quickly because it results in 'the Bends', which involves the formation of nitrogen bubbles in the blood and is potentially fatal.
If that's the case, why not use pure oxygen gas in SCUBA tanks? It seems like a good idea since it would

a) Enable divers to stay underwater for longer periods of time (I keep hearing that ordinary SCUBA tanks only give divers a pathetic hour or so of time underwater.
b) Possibly eliminate the chances of developing 'the Bends' upon surfacing. Well, it seems plausible, that is if the diver were to take a 10 minute deep-breathing session with pure oxygen to flush out whatever nitrogen's there in his lungs before hooking up a cylinder of pure oxygen and going for a dive. So if there's no gaseous nitrogen in his lungs and blood, then he wouldn't have to worry about nitrogen bubbles developing in his system.

Now those two possible advantages aren't hard to overlook, but since no one fills SCUBA tanks with pure oxygen, there must be some reason that I've overlooked, that discourages divers from filling the tanks with pure oxygen. So what is it?
Also, I hear that the oxygen cylinders used in hospitals have very high concentrations of oxygen; heck, there's one method of treatment called the Hyperbaric Oxygen Therapy (HBOT) where they give patients 100% pure oxygen at elevated pressures.
Hence I doubt whether the increase in pressure associated with diving is the problem here. So I reiterate:
Why is it a bad idea for divers to breathe pure oxygen underwater?

I guess most of the recent answers have kinda missed a main point, so I'll rephrase the question:
Why is it a bad idea for divers to breathe pure oxygen underwater? If it is indeed due to pressure considerations as most sources claim, then why doesn't it seem to be a problem when patients are given 100% pure oxygen in cases like the HBOT (which is performed at elevated pressures) ?

The other answers here, describing oxygen toxicity are telling what can go wrong if you have too much oxygen, but they are not describing two important concepts that should appear with their descriptions. Also, there is a basic safety issue with handling pressure tanks of high oxygen fraction.
An important property of breathed oxygen is its partial pressure. At normal conditions at sea level, the partial pressure of oxygen is about 0.21 atm. This is compatible with the widely known estimate that the atmosphere is about 78% nitrogen, 21% oxygen, and 1% "other". Partial pressures are added to give total pressure; this is Dalton's Law. As long as you don't use toxic gasses, you can replace the nitrogen and "other" with other gasses, like Helium, as long as you keep the partial pressure of oxygen near 0.21, and breathe the resulting mixtures without adverse effects.
There are two hazards that can be understood by considering the partial pressure of oxygen. If the partial pressure drops below about 0.16 atm, a normal person experiences hypoxia. This can happen by entering a room where oxygen has been removed. For instance, entering a room which has a constant source of nitrogen constantly displacing the room air, lowering the concentration -- and partial pressure -- of oxygen. Another way is to go to the tops of tall mountains. The total atmospheric pressure is lowered and the partial pressure of oxygen can be as low as 0.07 atm (summit of Mt. Everest) which is why very high altitude climbing requires carrying additional oxygen. Yet a third way is "horsing around" with Helium tanks -- repeatedly inhaling helium to produce very high pitched voices deprives the body of oxygen and the partial pressure of dissolved oxygen in the body falls, perhaps leading to loss of consciousness.
Alternatively, if the partial pressure rises above about 1.4 atm, a normal person experiences hyperoxia which can lead to oxygen toxicity (described in the other answers). At 1.6 atm the risk of central nervous system oxygen toxicity is very high. So, don't regulate the pressure that high? There's a problem. If you were to make a 10-foot long snorkel and dive to the bottom of a swimming pool to use it, you would fail to inhale. The pressure of air at your mouth would be about 1 atm, because the 10-foot column of air in the snorkel doesn't weigh very much. The pressure of water trying to squeeze the air out of you (like a tube of toothpaste) is about 1.3 atm. Your diaphragm is not strong enough to overcome the squeezing and fill your lungs with air. Divers overcome this problem by using a regulator (specifically, a demand valve), which allows the gas pressure at the outlet to be very near that of the ambient pressure. The principle job of the regulator is to reduce the very high pressure inside the tank to a much lower pressure at the outlet. The demand valve tries to only supply gas when the diver inhales and tries to supply it at very nearly ambient pressure. Notice that at depth the ambient pressure can be much greater than 1 atm, increasing by about 1 atm per 10 m (or 33 feet). If the regulator were to supply normal air at 2 atm pressure, the partial pressure of oxygen would be 0.42 atm. If at 3 atm, 0.63 atm. So as a diver descends, the partial pressure of oxygen automatically increases as a consequence of having to increase the gas pressure to allow the diver to inflate their lungs. Around 65 m (220 ft), the partial pressure of oxygen in an "air mix" would be high enough to risk hyperoxia and other dangerous consequences.
Now imagine a gas cylinder containing 100% oxygen. If we breathe from it at the surface, the partial pressure of oxygen is 1 atm -- high, but not dangerous. At a depth of 10 m, the partial pressure of supplied oxygen is 2 atm -- exceeding acceptable exposure limits. This is a general pattern -- raising the oxygen fraction of diving gasses decreases the maximum diving depth.
And you can't lower the partial pressure much because the lower limit, 0.16 atm, isn't that much lower than the 0.21 atm of sea level atmosphere.
One general category of solutions is to change gas mixes at various depths. This is complicated, requires a great deal of planning, and is outside the scope of your question. But it is certainly not as straightforward as just simplifying the gas mixtures or just raising the partial pressure of oxygen.
Additionally, compressed oxygen is a relatively annoying gas to work with. It is not itself flammable, but it makes every nearby organic thing flammable. For instance using grease or oil on or near an oxygen fitting risks spontaneously igniting the grease or oil. Merely having grease on your hand while handling oxygen refilling gear (with a small leak) can burn your hand.


Question:
Does anyone know of a freeware SPICE / circuit simulator?

SPICE (Simulation Program with Integrated Circuit Emphasis) is a general-purpose, open source analog electronic circuit simulator. It is a powerful program that is used in integrated circuit and board-level design to check the integrity of circuit designs and to predict circuit behavior. Wikipedia

ngSpice is available for gEDA.
gnuCAP is also available for gEDA.
LTSpice is free from Linear Technology.

I thought that one of the other analog chip makers had a spice too but I can't remember
who :(
I have been to a few talks on simulation given by physicists and EEs who have done
chip design. Each of the talks seems to end like this ---

Except for simple circuits you will spend most of your time getting models
    and determining where the models need to be modified for your application.
Unless you are doing work for an IC manufacturer the manufacturer will not
    give you detailed models.
You will not be able to avoid a prototype.
You should only simulate subsections of your design. Simulating the entire
    design is not usually practical.

Also most of the free simulators are not distributed with models. Re-distribution of
the models is usually a copyright violation. LTspice is distributed with models of
the Linear Tech parts. I am not sure the quality of the models. Most manufacturers
do not want to reveal too many details about their process.


Question:
The sensitive plant (Mimosa pudica) is a remarkable little plant whose characteristic feature is its ability to droop its leaves when disturbed:

Apparently, this ability to droop rests on the cells in the leaves of the sensitive plant being able to draw water out of themselves through changes in intracellular ion concentrations, which makes the leaves less turgid.
What I'm hazy about is how the plant "senses" vibrations. Plants don't really have a nervous system to speak of; how then does the sensitive plant "know" to droop when disturbed?

In fact, the idea of a plant nervous system is quite serious and constantly developing; of course those are rather local, simple signal pathways rather than an "animalian" centralized global network, but they use similar mechanisms -- depolarisation waves, neurotransmitter-like compounds, specialized cells... Here is a review paper by Brenner et al.
In the case of Mimosa, there is a good paper summing up Takao Sibaoka's long research of the topic.
In short, it seems that its petioles' phloem has cells which have polarized membranes and can trigger depolarization due to a mechanical stimulation. The signal then propagates to the corresponding pulvinus by a mixture of electrical and Cl- depolarization waves.
In the pulvinus, this signal triggers a second depolarization which coordinates the pulvinus' cells to trigger water pumping responsible for the leaf drop.
The transmission to the adjacent leaves is most likely mechanical, i.e. the movement of one dropping leaf excites another.
References:

 Brenner ED, Stahlberg R, Mancuso S, Vivanco J, Baluska F, Van Volkenburgh E. 2006. Plant neurobiology: an integrated view of plant signaling. Trends in plant science 11: 413–9.
 Sibaoka T. 1991. Rapid plant movements triggered by action potentials. The Botanical Magazine Tokyo 104: 73–95.


Question: For a molecule to have a smell it's necessary that the molecule be volatile enough to be in the air. So I think that excludes molecules which are solid at room temperature and atmospheric pressure. Maybe the question then is equivalent to: what is the highest molecular weight organic compound which is liquid at room temperature and atmospheric pressure?

I'll quote from $\ce{[1]}$:

The general requirements for an odorant are that it should be
volatile, hydrophobic and have a molecular weight less than
approximately 300 daltons. Ohloff (1994) has stated that the largest
known odorant is a labdane with a molecular weight of 296. The first
two requirements make physical sense, for the molecule has to reach
the nose and may need to cross membranes. The size requirement appears
to be a biological constraint. To be sure, vapor pressure (volatility)
falls rapidly with molecular size, but that cannot be the reason why
larger molecules have no smell, since some of the strongest odorants
(e.g. some steroids) are large molecules. In addition, the cut-off is
very sharp indeed: for example, substitution of the slightly larger
silicon atom for a carbon in a benzenoid musk causes it to become
odorless (Wrobel and Wannagat, 1982d).
A further indication that the size limit has something to do with the
chemoreception mechanism comes from the fact that specific anosmias
become more frequent as molecular size increases. At the “ragged edge”
of the size limit, subjects become anosmic to large numbers of
molecules. An informal poll among perfumers, for example has elicited
the fact that most of them are completely anosmic to one or more musks
(e.g. Galaxolide® mw 244.38) or, less commonly, ambergris odorants
such as Ambrox®, or the larger esters of salicylic acid.
One can probably infer from this that the receptors cannot accommodate
molecules larger than a certain size, and that this size is
genetically determined (Whissel-Buechy and Amoore, 1973) and varies
from individual to individual.

N.B.: Labdane's molecular formula is $\ce{C20H38}$, which gives a molecular weight (MW) of $\pu{278.5 Da}$ (Da). $\ce{[5]}$  Thus either the $\pu{296 Da}$ value is a typo, or the authors were quoting the MW of a labdane derivative.
Note added in response to answer posted by John Cuthbert (which was a nice find!):
While iodoform, at $\pu{394 Da}$, does indeed exceed the $\pu{>300 Da}$ "general requirement" provided above by Turin & Yoshii, a comparison of its estimated density to that of, e.g., labdane, indicates it's a much smaller molecule (iodoform's three iodine atoms add a lot of mass without a lot of size, at least relative to carbon, hydrogen, and oxygen):
I couldn't find labdane's density, but I found the density of one of its diols (i.e., labdane with an $\text{–OH}$ substituted for $\text{–H}$ in two places). So if we use its density, along with labane's molecular weight, we obtain:
$\pu{MW = 278.5 Da}$, $\pu{\rho = 0.9 g/cm^3}$ $\ce{[6]}$
=> estimated molecular volume ≈ $\pu{510 Å^3}$
Iodoform:
$\pu{MW = 393.732 Da}$, $\pu{\rho = 4.008 g/cm^3}$ $\ce{[7]}$
=> estimated molecular volume ≈ $\pu{160 Å^3}$
Even if the density of labdane were, say, 20% higher than that of the diol, we'd get a molecular volume of ≈ $\pu{430 Å^3}$, which is still far above that of iodoform.
This makes it clear that the limiting attribute is physical size rather than molecular weight, and that Turin & Yoshii were using molecular weight as a shorthand surrogate for size. This works reasonably well when comparing oxygenated hydrocarbons, but obviously breaks down when the compounds contain significantly heavier nuclei.  As Turin & Yoshii write more precisely at the end of the quoted passage:  "One can probably infer from this that the receptors cannot accommodate molecules larger than a certain size." [Emphasis mine.]
References
$\ce{[1]}$: "Structure-odor relationships: a modern perspective", by Luca Turin (Dept of Physiology, University College London, UK) and Fumiko Yoshii (Graduate School of Science and Technology, Niigata University, Japan), which appears as chapter 13 of: Handbook of Olfaction and Gustation. Richard L. Doty (ed.). 2nd ed., Marcel Dekker, 2003.
$\ce{[2]}$: Ohloff, G. Scent and fragrances: the fascination of odors and their chemical perspectives. Berlin, Springer, 1994.
$\ce{[3]}$: Wrobel D, Wannagat U. SILA PERFUMES. 2. SILALINALOOL. Chemischer Informationsdienst. 13(30), Jul 27, 1982.
$\ce{[4]}$: Whissell-Buechy D, Amoore JE. Letter: Odour-blindness to musk: simple recessive inheritance. Nature, 245(5421):157-8, Sep 21, 1973.
$\ce{[5]}$:
$\ce{[6]}$:
$\ce{[7]}$:


Question:
In single-cell RNA-seq data we have an inflated number of 0 (or near-zero) counts due to low mRNA capture rate and other inefficiencies.
How can we decide which genes are 0 due to gene dropout (lack of measurement sensitivity), and which are genuinely not expressed in the cell?
Deeper sequencing does not solve this problem as shown on the below saturation curve of 10x Chromium data:

Also see Hicks et al. (2017) for a discussion of the problem:

Zero can arise in two ways:

the gene was not expressing any RNA (referred to as structural zeros) or
the RNA in the cell was not detected due to limitations of current experimental protocols (referred to as dropouts)

Actually this is one of the main problems you have when analyzing scRNA-seq data, and there is no established method for dealing with this. Different (dedicated) algorithms deal with it in different ways, but mostly you rely on how good the error modelling of your software is (a great read is the review by Wagner, Regev & Yosef, esp. the section on "False negatives and overamplification"). There are a couple of options:

You can impute values, i.e. fill in the gaps on technical zeros. CIDR and scImpute do it directly. MAGIC and ZIFA project cells into a lower-dimensional space and use their similarity there to decide how to fill in the blanks.
Some people straight up exclude genes that are expressed in very low numbers. I can't give you citations off the top of my head, but many trajectory inference algorithms like monocle2 and SLICER have heuristics to choose informative genes for their analysis.
If the method you use for analysis doesn't model gene expression explicitly but uses some other distance method to quantify similarity between cells (like cosine distance, euclidean distance, correlation), then the noise introduced by dropout can be covered by the signal of genes that are highly expressed. Note that this is dangerous, as genes that are highly expressed are not necessarily informative.
ERCC spike ins can help you reduce technical noise, but I am not familiar with the Chromium protocol so maybe it doesn't apply there (?)

since we are speaking about noise, you might consider using a protocol with unique molecular identifiers. They remove the amplification errors almost completely, at least for the transcripts that you capture...
EDIT: Also, I would highly recommend using something more advanced than PCA to do the analysis. Software like the above-mentioned Monocle or destiny is easy to operate and increases the power of your analysis considerably.


Question:
I've just finished a Classical Mechanics course, and looking back on it some things are not quite clear. In the first half we covered the Lagrangian formalism, which I thought was pretty cool. I specially appreciated the freedom you have when choosing coordinates, and the fact that you can basically ignore constraint forces. Of course, most simple situations you can solve using good old $F=ma$, but for more complicated stuff the whole formalism comes in pretty handy.
Then in the second half we switched to Hamiltonian mechanics, and that's where I began to lose sight of why we were doing things the way we were. I don't have any problem understanding the Hamiltonian, or Hamilton's equations, or the Hamilton-Jacobi equation, or what have you. My issue is that I don't understand why would someone bother developing all this to do the same things you did before but in a different way. In fact, in most cases you need to start with a Lagrangian and get the momenta from $p = \frac{\partial L}{\partial \dot{q}}$, and the Hamiltonian from $H = \sum \dot{q_i}p_i - L$. But if you already have the Lagrangian, why not just solve the Euler-Lagrange equations?
I guess maybe there are interesting uses of the Hamiltion formalism and we just didn't do a whole lot of examples (it was the harmonic oscillator the whole way, pretty much). I've also heard that it allows a somewhat smooth transition into quantum mechanics. We did work out a way to get Schrödinger's equation doing stuff with the action. But still something's not clicking.
My questions are the following: Why do people use the Hamiltonian formalism? Is it better for theoretical work? Are there problems that are more easily solved using Hamilton's mechanics instead of Lagrange's? What are some examples of that?

There are several reasons for using the Hamiltonian formalism:

Statistical physics.  The standard thermal states weight of pure states is given according to
$$\text{Prob}(\text{state}) \propto e^{-H(\text{state})/k_BT}$$
So you need to understand Hamiltonians to do stat mech in real generality.

Geometrical prettiness.  Hamilton's equations say that flowing in time is equivalent to flowing along a vector field on phase space.  This gives a nice geometrical picture of how time evolution works in such systems.  People use this framework a lot in dynamical systems, where they study questions like 'is the time evolution chaotic?'.

The generalization to quantum physics.  The basic formalism of quantum mechanics (states and observables) is an obvious generalization of the Hamiltonian formalism.  It's less obvious how it's connected to the Lagrangian formalism, and way less obvious how it's connected to the Newtonian formalism.

[Edit in response to a comment:]
This might be too brief, but the basic story goes as follows:
In Hamiltonian mechanics, observables are elements of a commutative algebra which carries a Poisson bracket $\{\cdot,\cdot\}$.   The algebra of observables has a distinguished element, the Hamiltonian, which defines the time evolution via $d\mathcal{O}/dt = \{\mathcal{O},H\}$.  Thermal states are simply linear functions on this algebra.  (The observables are realized as functions on the phase space, and the bracket comes from the symplectic structure there.  But the algebra of observables is what matters:  You can recover the phase space from the algebra of functions.)
On the other hand, in quantum physics, we have an algebra of observables which is not commutative.  But it still has a bracket $\{\cdot,\cdot\} = -\frac{i}{\hbar}[\cdot,\cdot]$ (the commutator), and it still gets its time evolution from a distinguished element $H$, via $d\mathcal{O}/dt = \{\mathcal{O},H\}$.   Likewise, thermal states are still linear functionals on the algebra.


Question:
In evaluating the quality of a piece of software you are about to use (whether it's something you wrote or a canned package) in computational work, it is often a good idea to see how well it works on standard data sets or problems. Where might one obtain these tests for verifying computational routines?
(One website/book per answer, please.)

If you are interested in conducting an analysis on sparse matrices, I would also consider Davis's University of Florida Sparse Matrix Collection and the Matrix Market.


Question:
I haven't yet gotten a good answer to this: If you have two rays of light of the same wavelength and polarization (just to make it simple for now, but it easily generalizes to any range and all polarizations) meet at a point such that they're 180 degrees out of phase (due to path length difference, or whatever), we all know they interfere destructively, and a detector at exactly that point wouldn't read anything.
So my question is, since such an insanely huge number of photons are coming out of the sun constantly, why isn't any photon hitting a detector matched up with another photon that happens to be exactly out of phase with it? If you have an enormous number of randomly produced photons traveling random distances (with respect to their wavelength, anyway), that seems like it would happen, similar to the way that the sum of a huge number of randomly selected 1's and -1's would never stray far from 0. Mathematically, it would be:
$$\int_0 ^{2\pi} e^{i \phi} d\phi = 0$$
Of course, the same would happen for a given polarization, and any given wavelength.
I'm pretty sure I see the sun though, so I suspect something with my assumption that there are effectively an infinite number of photons hitting a given spot is flawed... are they locally in phase or something?

First let's deal with a false assumption:

similar to the way that the sum of a huge number of randomly selected 1's and -1's would never stray far from 0.

Suppose we have a set of $N$ random variables $X_i$, each independent and with equal probability of being either $+1$ or $-1$. Define
$$ S = \sum_{i=1}^N X_i. $$
Then, yes, the expectation of $S$ may be $0$,
$$ \langle S \rangle = \sum_{i=1}^N \langle X_i \rangle = \sum_{i=1}^N \left(\frac{1}{2}(+1) + \frac{1}{2}(-1)\right) = 0, $$
but the fluctuations can be significant. Since we can write
$$ S^2 = \sum_{i=1}^N X_i^2 + 2 \sum_{i=1}^N \sum_{j=i+1}^N X_i X_j, $$
then more manipulation of expectation values (remember, they always distribute over sums; also the expectation of a product is the product of the expectations if and only if the factors are independent, which is the case for us for $i \neq j$) yields
$$ \langle S^2 \rangle = \sum_{i=1}^N \langle X_i^2 \rangle + 2 \sum_{i=1}^N \sum_{j=i+1}^N \langle X_i X_j \rangle = \sum_{i=1}^N \left(\frac{1}{2}(+1)^2 + \frac{1}{2}(-1)^2\right) + 2 \sum_{i=1}^N \sum_{j=i+1}^N (0) (0) = N. $$
The standard deviation will be
$$ \sigma_S = \left(\langle S^2 \rangle - \langle S \rangle^2\right)^{1/2} = \sqrt{N}. $$
This can be arbitrarily large. Another way of looking at this is that the more coins you flip, the less likely you are to be within a fixed range of breaking even.

Now let's apply this to the slightly more advanced case of independent phases of photons. Suppose we have $N$ independent photons with phases $\phi_i$ uniformly distributed on $(0, 2\pi)$. For simplicity I will assume all the photons have the same amplitude, set to unity. Then the electric field will have strength
$$ E = \sum_{i=1}^N \mathrm{e}^{\mathrm{i}\phi_i}. $$
Sure enough, the average electric field will be $0$:
$$ \langle E \rangle = \sum_{i=1}^N \langle \mathrm{e}^{\mathrm{i}\phi_i} \rangle = \sum_{i=1}^N \frac{1}{2\pi} \int_0^{2\pi} \mathrm{e}^{\mathrm{i}\phi}\ \mathrm{d}\phi = \sum_{i=1}^N 0 = 0. $$
However, you see images not in electric field strength but in intensity, which is the square-magnitude of this:
$$ I = \lvert E \rvert^2 = \sum_{i=1}^N \mathrm{e}^{\mathrm{i}\phi_i} \mathrm{e}^{-\mathrm{i}\phi_i} + \sum_{i=1}^N \sum_{j=i+1}^N \left(\mathrm{e}^{\mathrm{i}\phi_i} \mathrm{e}^{-\mathrm{i}\phi_j} + \mathrm{e}^{-\mathrm{i}\phi_i} \mathrm{e}^{\mathrm{i}\phi_j}\right) = N + 2 \sum_{i=1}^N \sum_{j=i+1}^N \cos(\phi_i-\phi_j). $$
Paralleling the computation above, we have
$$ \langle I \rangle = \langle N \rangle + 2 \sum_{i=1}^N \sum_{j=i+1}^N \frac{1}{(2\pi)^2} \int_0^{2\pi}\!\!\int_0^{2\pi} \cos(\phi-\phi')\ \mathrm{d}\phi\ \mathrm{d}\phi' = N + 0 = N. $$
The more photons there are, the greater the intensity, even though there will be more cancellations.

So what does this mean physically? The Sun is an incoherent source, meaning the photons coming from its surface really are independent in phase, so the above calculations are appropriate. This is in contrast to a laser, where the phases have a very tight relation to one another (they are all the same).
Your eye (or rather each receptor in your eye) has an extended volume over which it is sensitive to light, and it integrates whatever fluctuations occur over an extended time (which you know to be longer than, say, $1/60$ of a second, given that most people don't notice faster refresh rates on monitors). In this volume over this time, there will be some average number of photons. Even if the volume is small enough such that all opposite-phase photons will cancel (obviously two spatially separated photons won't cancel no matter their phases), the intensity of the photon field is expected to be nonzero.
In fact, we can put some numbers to this. Take a typical cone in your eye to have a diameter of $2\ \mathrm{µm}$, as per Wikipedia. About $10\%$ of the Sun's $1400\ \mathrm{W/m^2}$ flux is in the $500\text{–}600\ \mathrm{nm}$ range, where the typical photon energy is $3.6\times10^{-19}\ \mathrm{J}$. Neglecting the effects of focusing among other things, the number of photons in play in a single receptor is something like
$$ N \approx \frac{\pi (1\ \mathrm{µm})^2 (140\ \mathrm{W/m^2}) (0.02\ \mathrm{s})}{3.6\times10^{-19}\ \mathrm{J}} \approx 2\times10^7. $$ The fractional change in intensity from "frame to frame" or "pixel to pixel" in your vision would be something like $1/\sqrt{N} \approx 0.02\%$. Even give or take a few orders of magnitude, you can see that the Sun should shine steadily and uniformly.


Question:
A few years ago, MapReduce was hailed as revolution of distributed programming. There have also been critics but by and large there was an enthusiastic hype. It even got patented! [1]
The name is reminiscent of map and reduce in functional programming, but when I read (Wikipedia)

Map step: The master node takes the input, divides it into smaller sub-problems, and distributes them to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes the smaller problem, and passes the answer back to its master node.
Reduce step: The master node then collects the answers to all the sub-problems and combines them in some way to form the output – the answer to the problem it was originally trying to solve.

or [2]

Internals of MAP: [...] MAP splits up the input value into words. [...] MAP is meant to associate each given key/value pair of the input with potentially many intermediate key/value pairs.
Internals of REDUCE: [...] [REDUCE] performs imperative aggregation (say, reduction): take many values, and reduce them to a single value.

I can not help but think: this is divide & conquer (in the sense of Mergesort), plain and simple! So, is there (conceptual) novelty in MapReduce somewhere, or is it just a new implementation of old ideas useful in certain scenarios?

 US Patent 7,650,331: "System and method for efficient large-scale data processing " (2010)
Google’s MapReduce programming model — Revisited by R. Lämmel (2007)

I can not help but think: this is divide & conquer, plain and simple!

M/R is not divide & conquer. It does not involve the repeated application of an algorithm to a smaller subset of the previous input.  It's a pipeline (a function specified as a composition of simpler functions) where pipeline stages are alternating map and reduce operations.  Different stages can perform different operations.

So, is there (conceptual) novelty in MapReduce somewhere, or is it just a new implementation of old ideas useful in certain scenarios?

MapReduce does not break new ground in the theory of computation -- it does not show a new way of decomposing a problem into simpler operations.  It does show that particular simpler operations are practical for a particular class of problem.

The MapReduce paper's contribution was

evaluating a pipeline of two well understood orthogonal operators that can be distributed efficiently and fault-tolerantly on a particular problem: creating a text index of large corpus
benchmarking map-reduce on that problem to show how much data is transferred between nodes and how latency differences in stages affect overall latency
showing how to make the system fault tolerant so machine failures during computation can be compensated for automatically
identifying specific useful implementation choices and optimizations

Some of the critiques fall into these classes:

"Map/reduce does not break new ground in theory of computation."  True.  The original paper's contribution was that these well-understood operators with a specific set of optimizations had been successfully used to solve real problems more easily and fault-tolerantly than one-off solutions.
"This distributed computation doesn't easily decompose into map & reduce operations".  Fair enough, but many do.
"A pipeline of n map/reduce stages require latency proportional to the number of reduce steps of the pipeline before any results are produced."  Probably true.  The reduce operator does have to receive all its input before it can produce a complete output.
"Map/reduce is overkill for this use-case."  Maybe.  When engineers find a shiny new hammer, they tend to go looking for anything that looks like a nail.  That doesn't mean that the hammer isn't a well-made tool for a certain niche.
"Map/reduce is a poor replacement for a relational DB."  True.  If a relational DB scales to your data-set then wonderful for you -- you have options.


Question:
I've implemented a gaussian blur fragment shader in GLSL. I understand the main concepts behind all of it: convolution, separation of x and y using linearity, multiple passes to increase radius...
I still have a few questions though:

What's the relationship between sigma and radius?
I've read that sigma is equivalent to radius, I don't see how sigma is expressed in pixels. Or is "radius" just a name for sigma, not related to pixels?

How do I choose sigma?
Considering I use multiple passes to increase sigma, how do I choose a good sigma to obtain the sigma I want at any given pass? If the resulting sigma is equal to the square root of the sum of the squares of the sigmas and sigma is equivalent to radius, what's an easy way to get any desired radius?

What's the good size for a kernel, and how does it relate to sigma?
I've seen most implementations use a 5x5 kernel. This is probably a good choice for a fast implementation with decent quality, but is there another reason to choose another kernel size? How does sigma relate to the kernel size? Should I find the best sigma so that coefficients outside my kernel are negligible and just normalize?

What's the relationship between sigma and radius? I've read that sigma is equivalent to radius, I don't see how sigma is expressed in pixels. Or is "radius" just a name for sigma, not related to pixels?

There are three things at play here. The variance, ($\sigma^2$), the radius, and the number of pixels. Since this is a 2-dimensional gaussian function, it makes sense to talk of the covariance matrix $\boldsymbol{\Sigma}$ instead. Be that as it may however, those three concepts are weakly related.
First of all, the 2-D gaussian is given by the equation:
$$
g({\bf z}) = \frac{1}{\sqrt{(2 \pi)^2 |\boldsymbol{\Sigma}|}} e^{-\frac{1}{2} ({\bf z}-\boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} \ ({\bf z}-\boldsymbol{\mu})}
$$
Where ${\bf z}$ is a column vector containing the $x$ and $y$ coordinate in your image. So, ${\bf z} = \begin{bmatrix} x \\ y\end{bmatrix}$, and $\boldsymbol{\mu}$ is a column vector codifying the mean of your gaussian function, in the $x$ and $y$ directions $\boldsymbol{\mu} = \begin{bmatrix} \mu_x \\ \mu_y\end{bmatrix}$.
Example:
Now, let us say that we set the covariance matrix $\boldsymbol{\Sigma} = \begin{bmatrix} 1   & 0 \\ 0 & 1\end{bmatrix}$, and $\boldsymbol{\mu} = \begin{bmatrix} 0 \\ 0\end{bmatrix}$. I will also set the number of pixels to be $100$ x $100$. Furthermore, my 'grid', where I evaluate this PDF, is going to be going from $-10$ to $10$, in both $x$ and $y$. This means I have a grid resolution of $\frac{10 - (-10)}{100} = 0.2$. But this is completely arbitrary. With those settings, I will get the probability density function image on the left. Now, if I change the 'variance', (really, the covariance), such that $\boldsymbol{\Sigma} = \begin{bmatrix} 9   & 0 \\ 0 & 9\end{bmatrix}$ and keep everything else the same, I get the image on the right.

The number of pixels are still the same for both, $100$ x $100$, but we changed the variance. Suppose instead we do the same experiment, but use $20$ x $20$ pixels instead, but I still ran from $-10$ to $10$. Then, my grid has a resolution of $\frac{10-(-10)}{20} = 1$. If I use the same covariances as before, I get this:

These are how you must understand the interplay between those variables. If you would like the code, I can post that here as well.

How do I choose sigma?

The choice of the variance/covariance-matrix of your gaussian filter is extremely application dependent. There is no 'right' answer. That is like asking what bandwidth should one choose for a filter. Again, it depends on your application. Typically, you want to choose a gaussian filter such that you are nulling out a considerable amount of high frequency components in your image. One thing you can do to get a good measure, is compute the 2D DFT of your image, and overlay its co-efficients with your 2D gaussian image. This will tell you what co-efficients are being heavily penalized.
For example, if your gaussian image has a covariance so wide that it is encompassing many high frequency coefficients of your image, then you need to make its covariance elements smaller.


Question:
I've seen lots of schematics use \$V_{CC}\$ and \$V_{DD}\$ interchangeably.

I know \$V_{CC}\$ and \$V_{DD}\$ are for positive voltage, and \$V_{SS}\$ and \$V_{EE}\$ are for ground, but what is the difference between each of the two?
Do the \$C\$, \$D\$, \$S\$, and \$E\$ stand for something?

For extra credit: Why \$V_{DD}\$ and not simply \$V_D\$?

Back in the pleistoscene (1960s or earlier), logic was implemented with bipolar transistors.  Even more specifically, they were NPN because for some reasons I'm not going to get into, NPN were faster.  Back then it made sense to someone that the positive supply voltage would be called Vcc where the "c" stands for collector.  Sometimes (but less commonly) the negative supply was called Vee where "e" stands for emitter.
When FET logic came about, the same kind of naming was used, but now the positive supply was Vdd (drain) and the negative Vss (source).  With CMOS this makes no sense, but it persists anyway.  Note that the "C" in CMOS stands for "complementary".  That means both N and P channel devices are used in about equal numbers.  A CMOS inverter is just a P channel and a N channel MOSFET in its simplest form.  With roughly equal numbers of N and P channel devices, drains aren't more likely to be positive than sources, and vice versa.  However, the Vdd and Vss names have stuck for historical reasons.  Technically Vcc/Vee is for bipolar and Vdd/Vss for FETs, but in practise today Vcc and Vdd mean the same, and Vee and Vss mean the same.


Question:
The purpose of this question is to ask about the role of mathematical rigor in physics. In order to formulate a question that can be answered, and not just discussed, I divided this large issue into five specific questions.
Update February, 12, 2018: Since the question was put yesterday on hold as too board, I ask future to refer only to questions one and two listed below. I will ask a separate questions on item 3 and 4. Any information on question 5 can be added as a remark.

What are the most important and the oldest insights (notions, results) from physics that are still lacking rigorous mathematical
  formulation/proofs.
The endeavor of rigorous mathematical explanations, formulations, and proofs for notions and results from physics is mainly taken by
  mathematicians. What are examples that this endeavor was beneficial to
  physics itself.

What are examples that insisting on rigour delayed progress in physics.
What are examples that solid mathematical understanding of certain issues from physics came from further developments in physics itself. (In particular, I am interested in cases where mathematical rigorous understanding of issues from classical mechanics required quantum mechanics, and also in cases where progress in physics was crucial to rigorous mathematical solutions of questions in mathematics not originated in physics.)
The role of rigor is intensely discussed in popular books and blogs. Please supply references (or better annotated references) to academic studies of the role of mathematical rigour in modern physics.

(Of course, I will be also thankful to answers which elaborate on a single item related to a single question out of these five questions. See update)
Related Math Overflow questions:

Examples-of-non-rigorous-but-efficient-mathematical-methods-in-physics (related to question 1);
Examples-of-using-physical-intuition-to-solve-math-problems;
Demonstrating-that-rigour-is-important.

Rigorous arguments are very similar to computer programming--- you need to write a proof which can (in principle) ultimately be carried out in a formal system. This is not easy, and requires defining many data-structures (definitions), and writing many subroutines (lemmas), which you use again and again. Then you prove many results along the way, only some of which are of general usefulness.
This activity is extremely illuminating, but it is time consuming, and tedious, and requires a great deal of time and care. Rigorous arguments also introduce a lot of pedantic distinctions which are extremely important for the mathematics, but not so important in the cases one deals with in physics.
In physics, you never have enough time, and we must always have a only just precise enough understanding of the mathematics that can be transmitted maximally quickly to the next generation. Often this means that you forsake full rigor, and introduce notational short-cuts and imprecise terminology that makes turning the argument rigorous difficult.
Some of the arguments in physics though are pure magic. For me, the replica trick is the best example. If this ever gets a rigorous version, I will be flabbergasted.

1) What are the most important and the oldest insights (notions, results) from physics that are still lacking rigorous mathematical formulation/proofs.

Here are old problems which could benefit from rigorous analysis:

Mandelstam's double-dispersion relations: The scattering amplitude for 2 particle to 2 particle scattering can be analytically expanded as an integral over the imaginary discontinuity $\rho(s)$ in the s parameter, and then this discontinuity $\rho(s)$ can be written as an integral over the t parameter, giving a double-discontinuity $\rho(s,t)$ If you go the other way, expand the discontinuity in t first then in s, you get the same function. Why is that? It was argued from perturbation theory by Mandelstam, and there was some work in the 1960s and early 1970s, but it was never solved as far as I know.
The oldest, dating back centuries: Is the (Newtonian, comet and asteroid free) solar system stable for all time? This is a famous one. Rigorous bounds on where integrability fails will help. The KAM theorem might be the best answer possible, but it doesn't answer the question really, since you don't know whether the planetary perturbations are big enough to lead to instability for 8 planets some big moons, plus sun.
continuum statistical mechanics: What is a thermodynamic ensemble for a continum field? What is the continuum limit of a statistical distribution? What are the continuous statistical field theories here?
What are the generic topological solitonic solutions to classical nonlinear field equations? Given a classical equation, how do you find the possible topological solitons? Can they all be generated continuously from given initial data? For a specific example, consider the solar-plasma--- are there localized magneto-hydrodynamic solitons?

There are a bazillion problems here, but my imagination fails.

2) The endeavor of rigorous mathematical explanations, formulations, and proofs for notions and results from physics is mainly taken by mathematicians. What are examples that this endeavor was beneficial to physics itself.

There are a few examples, but I think they are rare:

Penrose's rigorous proof of the existence of singularities in a closed trapped surface is the canonical example: it was a rigorous argument, derived from Riemannian geometry ideas, and it was extremely important for clarifying what's going on in black holes.
Quasi-periodic tilings, also associated with Penrose, first arose in Hao and Wang's work in pure logic, where they were able to demonstrate that an appropriate tiling with complicated matching edges could do full computation. The number of tiles were reduced until Penrose gave only 2, and finally physicists discovered quasicrystals. This is spectacular, because here you start in the most esoteric non-physics part of pure mathematics, and you end up at the most hands-on of experimental systems.
Kac-Moody algebras: These came up in half-mathematics, half early string theory. The results became physical in the 1980s when people started getting interested in group manifold models.
The ADE classificiation from Lie group theory (and all of Lie group theory) in mathematics is essential in modern physics. Looking back further, Gell-Mann got SU(3) quark symmetry by generalizing isospin in pure mathematics.
Obstruction theory was essential in understanding how to formulate 3d topological field theories (this was the subject of a recent very interesting question), which have application in the fractional quantum hall effect. This is very abstract mathematics connected to laboratory physics, but only certain simpler parts of the general mathematical machinery are used.

3) What are examples that insisting on rigour delayed progress in physics.

This has happened several times, unfortunately.

Statistical mechanics: The lack of rigorous proof of Boltzmann ergodicity delayed the acceptance of the idea of statistical equilibrium. The rigorous arguments were faulty--- for example, it is easy to prove that there are no phase transitions in finite volume (since the Boltzmann distribution is analytic), so this was considered a strike against Boltzmann theory, since we see phase transitions. You could also prove all sorts of nonsense about mixing entropy (which was fixed by correctly dealing with classical indistinguishability). Since there was no proof that fields would come to thermal equilibrium, some people believed that blackbody light was not thermal. This delayed acceptance of Planck's theory, and Einstein's. Statistical mechanics was not fully accepted until Onsager's Ising model solution in 1941.
Path integrals: This is the most notorious example. These were accepted by some physicists immediately in the 1950s, although =the formalism wasn't at all close to complete until Candlin formulated Grassman variables in 1956. Past this point, they could have become standard, but they didn't. The formalism had a bad reputation for giving wrong results, mostly because people were uncomfortable with the lack of rigor, so that they couldn't trust the method. I heard a notable physicist complain in the 1990s that the phase-space path integral (with p and q) couldn't possibly be correct because p and q don't commute, and in the path integral they do because they are classical numbers (no, actually, they don't--- their value in an insertion depends discontinuously on their time order in the proper way). It wasn't until the early 1970s that physicists became completely comfortable with the method, and it took a lot of selling to overcome the resistance.
Quantum field theory construction: The rigorous methods of the 1960s built up a toolbox of complicated distributional methods and perturbation series resummation which turns out to be the least useful way of looking at the thing. It's now C* algebras and operator valued distributions. The correct path is through the path integral the Wilsonian way, and this is closer to the original point of view of Feynman and Schwinger. But a school of rigorous physicists in the 1960s erected large barriers to entry in field theory work, and progress in field theory was halted for a decade, until rigor was thrown out again in the 1970s. But a proper rigorous formulation of quantum fields is still missing.

In addition to this, there are countless no-go theorems that delayed the discovery of interesting things:

Time cannot be an operator (Pauli): this delayed the emergence of the path integral particle formulation due to Feynman and Schwinger. Here, the time variable on the particle-path is path-integrated just like anything else.
Von-Neumann's proof of no-hidden variables: This has a modern descendent in the Kochen Sprecher theorem about entangled sets of qubits. This delayed the Bohm theory, which faced massive resistance at first.
No charges which transform nontrivially under the Lorentz group(Coleman-Mandula): This theorem had both positive and negative implications. It killed SU(6) theories (good), but it made people miss supersymmetry (bad).
Quasicrystal order is impossible: This "no go" theorem is the standard proof that periodic order (the general definition of crystals) is restricted to the standard space-groups. This made quasicrystals bunk. The assumption that is violated is the assumption of strict periodicity.
No supergravity compactifications with chiral fermions (Witten): this theorem assumed manifold compactification, and missed orbifolds of 11d SUGRA, which give rise to the heterotic strings (also Witten, with Horava, so Witten solved the problem).

4) What are examples that solid mathematical understanding of certain issues from physics came from further developements in physics itself. (In particular, I am interested in cases where mathematical rigorous understanding of issues from classical mechanics required quantum mechenics, and also in cases where progress in physics was crucial to rigorous mathematical solutions of questions in mathematics not originated in physics.)

There are several examples here:

Understanding the adiabatic theorem in classical mechanics (that the action is an adiabatic invariant) came from quantum mechanics, since it was clear that it was the action that needed to be quantized, and this wouldn't make sense without it being adiabatic invariant. I am not sure who proved the adiabatic theorem, but this is exactly what you were asking for--- an insightful classical theorem that came from quantum mechanics (although some decades before modern quantum mechanics)
The understanding of quantum anomalies came directly from a physical observation (the high rate of neutral pion decay to two photons). Clarifying how this happens through Feynman diagrams, even though a naive argument says it is forbidden led to complete understanding of all anomalous terms in terms of topology. This in turn led to the development of Chern-Simons theory, and the connection with Knot polynomials, discovered by Witten, and earning him a Fields medal.
Distribution theory originated in Dirac's work to try to give a good foundation for quantum mechanics. The distributional nature of quantum fields was understood by Bohr and Rosenfeld in the 1930s, and the mathematics theory was essentially taken from physics into mathematics. Dirac already defined distributions using test functions, although I don't think he was pedantic about the test-function space properties.

5) The role of rigor is intensly discussed in popular books and blogs. Please supply references (or better annotated references) to academic studies of the role of mathematical rigour in modern physics.

I can't do this, because I don't know any. But for what it's worth, I think it's a bad idea to try to do too much rigor in physics (or even in some parts of mathematics). The basic reason is that rigorous formulations have to be completely standardized in order for the proofs of different authors to fit-together without seams, and this is only possible in very long hindsight, when the best definitions become apparent. In the present, we're always muddling through fog. So there is always a period where different people have slightly different definitions of what they mean, and the proofs don't quite work, and mistakes can happen. This isn't so terrible, so long as the methods are insightful.
The real problem is the massive barrier to entry presented by rigorous definitions. The actual arguments are always much less daunting than the superficial impression you get from reading the proof, because most of the proof is setting up machinery to make the main idea go through. Emphasizing the rigor can put undue emphasis on the machinery rather than the idea.
In physics, you are trying to describe what a natural system is doing, and there is no time to waste in studying sociology. So you can't learn all the machinery the mathematicians standardize on at any one time, you just learn the ideas. The ideas are sufficient for getting on, but they aren't sufficient to convince mathematicians you know what you're talking about (since you have a hard time following the conventions). This is improved by the internet, since the barriers to entry have fallen down dramatically, and there might be a way to merge rigorous and nonrigorous thinking today in ways that were not possible in earlier times.


Question:
I'm a mathematician who recently became very interested in questions related to mathematical physics but somehow, I  faced difficulties in penetrating the literature... I'd highly appreciate any help with the following question:
My aim is to relate a certain (equivariant) linear sigma model on a disc (with a non-compact target $\mathbb C$) as constructed in the exciting work of Gerasimov, Lebedev and Oblezin in Archimedean L-factors and Topological Field Theories I, to integrable systems (in the sense of Dubrovin, if you like).
More precisely, I'd like to know if it's possible to express "the" correlation function of an (equivariant) linear sigma model (with non-compact target) as in the above reference in terms of a $\tau$-function of an associated integrable system?
As far as I've understood from the literature, for a large class of related non-linear sigma models (or models like conformal topological field theories) such a translation can be done by translating the field theory (or at least some parts of it) into some Frobenius manifold (as in Dubrovin's approach, e.g., but other approaches are of course also welcome). Unfortunately, so far, I haven't been able to understand how to make things work in the setting of (equivariant) linear sigma models (with non-compact target).
Any help or hints would be highly appreciated!

This is a reference resources question, masquerading as an answer, given the constraints of the site. The question hardly belongs here, and has been duplicated in the overflow cousin site . It might well be deleted.
There have been schools and proceedings on the subject,
Integrability: From Statistical Systems to Gauge TheoryLecture Notes of the Les Houches Summer School: Volume 106, June  2016, Volume 106,
Patrick Dorey, Gregory Korchemsky, Nikita Nekrasov, Volker Schomerus, Didina Serban, and Leticia Cugliandolo. Print publication date: 2019,  ISBN-13: 9780198828150, Published to Oxford Scholarship Online: September 2019.
DOI: 10.1093/oso/9780198828150.001.0001
including, specifically,
Integrability in 2D fields theory/sigma-models, Sergei L Lukyanov &
Alexander B Zamolodchikov.
DOI:10.1093/oso/9780198828150.003.0006
Integrability in sigma-models, K. Zarembo.
DOI:10.1093/oso/9780198828150.003.0005

I am particular to
Integrable 2d sigma models: Quantum corrections to geometry from RG flow, Ben Hoare, Nat Levine, Arkady Tseytlin, Nucl Phys
B949 (2019) 114798 ,  but that's only by dint of personal connectivity...